[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:01 dist-test-slave-z2zg krb5kdc[6617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:01 dist-test-slave-z2zg krb5kdc[6617](info): set up 2 sockets
Sep 29 03:27:01 dist-test-slave-z2zg krb5kdc[6617](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 03:27:01.972607 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.390s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:02.086376 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42785
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:42785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:02.164535 6630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:02.164758 6630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:02.164793 6630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:02.165869 6630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:02.165931 6630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:02.165958 6630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:02.165987 6630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:02.166010 6630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:02.167455 6630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:42785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:02.167717 6630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:02.167925 6630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:02.172950 6636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.172906 6635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.172963 6630 server_base.cc:1034] running on GCE node
W20260929 03:27:02.172930 6638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.173274 6630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:02.173544 6630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:02.174692 6630 hybrid_clock.cc:648] HybridClock initialized: now 1790652422174679 us; error 29 us; skew 500 ppm
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:02.177050 6630 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:02.177497 6630 webserver.cc:460] Webserver started at http://127.6.116.190:43207/ using document root <none> and password file <none>
I20260929 03:27:02.177714 6630 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:02.177757 6630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:02.177902 6630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:02.178921 6630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "961a58fcd0c74e8dbf172e951ccbe1e3"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "eff681489ab91601051f6bd377bf3d13"
server_key_iv: "1c06fca8989954a63a81f45a8567e890"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.179352 6630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "961a58fcd0c74e8dbf172e951ccbe1e3"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "eff681489ab91601051f6bd377bf3d13"
server_key_iv: "1c06fca8989954a63a81f45a8567e890"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.180637 6630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:02.181316 6645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.181478 6630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:02.181576 6630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "961a58fcd0c74e8dbf172e951ccbe1e3"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "eff681489ab91601051f6bd377bf3d13"
server_key_iv: "1c06fca8989954a63a81f45a8567e890"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.181646 6630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:02.198417 6630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:02.200204 6630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:02.200358 6630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:02.204205 6630 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:42785
I20260929 03:27:02.204252 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:42785 every 8 connection(s)
I20260929 03:27:02.204624 6630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:02.205426 6698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:02.207700 6698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3: Bootstrap starting.
I20260929 03:27:02.208215 6698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:02.208437 6698 log.cc:824] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:02.209138 6698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3: No bootstrap required, opened a new log
I20260929 03:27:02.210598 6698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } }
I20260929 03:27:02.210732 6698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:02.210754 6698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961a58fcd0c74e8dbf172e951ccbe1e3, State: Initialized, Role: FOLLOWER
I20260929 03:27:02.210908 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 6630
I20260929 03:27:02.210909 6698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } }
I20260929 03:27:02.210981 6698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:02.211007 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:02.211059 6698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:02.211113 6698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:02.211642 6698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } }
I20260929 03:27:02.211687 6610 external_mini_cluster.cc:1428] Setting key c5dcab62b0933c2b2f3541f95d951739
I20260929 03:27:02.211733 6698 leader_election.cc:302] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 961a58fcd0c74e8dbf172e951ccbe1e3; no voters:
I20260929 03:27:02.211923 6698 leader_election.cc:288] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:02.211942 6701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:02.212049 6701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [term 1 LEADER]: Becoming Leader. State: Replica: 961a58fcd0c74e8dbf172e951ccbe1e3, State: Running, Role: LEADER
I20260929 03:27:02.212172 6698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:02.212145 6701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } }
I20260929 03:27:02.212592 6703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 961a58fcd0c74e8dbf172e951ccbe1e3. Latest consensus state: current_term: 1 leader_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } } }
I20260929 03:27:02.212674 6703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:02.212827 6702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "961a58fcd0c74e8dbf172e951ccbe1e3" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42785 } } }
I20260929 03:27:02.212877 6702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:02.212983 6709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:02.213462 6709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:02.215152 6709 catalog_manager.cc:1347] Generated new cluster ID: 5b972224c64c4aa2a752599409c73cc4
I20260929 03:27:02.215205 6709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:02.225895 6709 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:02.226478 6709 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:02.229645 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.213009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.6.116.190:42785)
0929 03:27:02.213297 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.213305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.213336 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:02.215523 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.215529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.215580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.215708 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.217792 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.219237 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.220704 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.221353 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.221452 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.225665 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.225670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.225670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.225681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.227715 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.228194 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.228197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.228198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.228301 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.228557 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.228558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.228558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.228826 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.228969 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.229268 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.229412 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":74,"threads_started":1}
I20260929 03:27:02.230521 6709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 961a58fcd0c74e8dbf172e951ccbe1e3: Generated new TSK 0
I20260929 03:27:02.230691 6709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:02.262588 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:02.365056 6724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:02.365257 6724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:02.365281 6724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:02.366436 6724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:02.366477 6724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:02.366504 6724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:02.367986 6724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:02.368219 6724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:02.368455 6724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:02.373255 6729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.373293 6732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.373293 6730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.373440 6724 server_base.cc:1034] running on GCE node
I20260929 03:27:02.373669 6724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:02.373951 6724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:02.375100 6724 hybrid_clock.cc:648] HybridClock initialized: now 1790652422375078 us; error 36 us; skew 500 ppm
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:02.377619 6724 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:02.378028 6724 webserver.cc:460] Webserver started at http://127.6.116.129:38985/ using document root <none> and password file <none>
I20260929 03:27:02.378257 6724 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:02.378301 6724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:02.378441 6724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:02.379515 6724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "65da06374d5d4ff7ac46dc24e1ef1a6c"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "c11c1c3baa8f18eedb7f2205604758bf"
server_key_iv: "c9f8ed4e448070633f97b5f867a06452"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.379874 6724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "65da06374d5d4ff7ac46dc24e1ef1a6c"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "c11c1c3baa8f18eedb7f2205604758bf"
server_key_iv: "c9f8ed4e448070633f97b5f867a06452"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.381145 6724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:02.381824 6739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.381989 6724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:02.382084 6724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "65da06374d5d4ff7ac46dc24e1ef1a6c"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "c11c1c3baa8f18eedb7f2205604758bf"
server_key_iv: "c9f8ed4e448070633f97b5f867a06452"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.382174 6724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:02.404196 6724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:02.406147 6724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:02.406312 6724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:02.406536 6724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:02.406838 6724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:02.406894 6724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.406929 6724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:02.406957 6724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.412600 6724 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:40281
I20260929 03:27:02.412649 6854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:40281 every 8 connection(s)
I20260929 03:27:02.412946 6724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:02.419453 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 6724
I20260929 03:27:02.419538 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:02.419710 6610 external_mini_cluster.cc:1428] Setting key eb36361180a532c4f155082f4a6d7295
I20260929 03:27:02.422694 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.413710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:42785 (local address 127.6.116.129:52421)
0929 03:27:02.413867 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:02.413907 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:02.414617 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:02.414869 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:02.414872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:02.415557 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:02.415741 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.415745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.416412 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.416412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.417199 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.417201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.417525 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.417525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.417616 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.418153 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:02.418169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:02.419665 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:02.421290 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.421297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.421299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.421613 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.421844 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.421845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.421845 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.421960 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.422241 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:02.422245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:02.422349 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 03:27:02.422428 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 03:27:02.422539 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260929 03:27:02.422689 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.413743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:52421 (local address 127.6.116.190:42785)
0929 03:27:02.413804 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.413806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.413983 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:27:02.414728 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.414730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.414744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.414816 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.415817 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.416321 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.417268 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.417455 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.417504 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.419746 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.419747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.419747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.419768 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.421209 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.421687 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.421688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.421688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.421760 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.422025 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.422026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.422026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.422166 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.422246 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.422472 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.422562 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:27:02.423501 6855 heartbeater.cc:347] Connected to a master server at 127.6.116.190:42785
I20260929 03:27:02.423599 6855 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:02.423777 6855 heartbeater.cc:511] Master 127.6.116.190:42785 requested a full tablet report, sending...
I20260929 03:27:02.424297 6662 ts_manager.cc:194] Registered new tserver with Master: 65da06374d5d4ff7ac46dc24e1ef1a6c (127.6.116.129:40281)
I20260929 03:27:02.424894 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:52421
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:02.445684 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:02.526065 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:02.526257 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:02.526298 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:02.527433 6860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:02.527472 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:02.527499 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:02.528991 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:02.529187 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:02.529353 6860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:02.534289 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.534269 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.534441 6860 server_base.cc:1034] running on GCE node
W20260929 03:27:02.534329 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.534713 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:02.535002 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:02.536214 6860 hybrid_clock.cc:648] HybridClock initialized: now 1790652422536182 us; error 34 us; skew 500 ppm
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:02.539260 6860 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:02.539742 6860 webserver.cc:460] Webserver started at http://127.6.116.130:34857/ using document root <none> and password file <none>
I20260929 03:27:02.539984 6860 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:02.540050 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:02.540194 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:02.541221 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "91bacae741254bf7be9cb0f64d95bc84"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "44104c6e4b5bbc72384be49d4f8e058f"
server_key_iv: "eacae7ef4304bb2009c3559c23728db7"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.541589 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "91bacae741254bf7be9cb0f64d95bc84"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "44104c6e4b5bbc72384be49d4f8e058f"
server_key_iv: "eacae7ef4304bb2009c3559c23728db7"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.542876 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:02.543675 6875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.543859 6860 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.543946 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "91bacae741254bf7be9cb0f64d95bc84"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "44104c6e4b5bbc72384be49d4f8e058f"
server_key_iv: "eacae7ef4304bb2009c3559c23728db7"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.544032 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:02.556922 6860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:02.558646 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:02.558805 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:02.559188 6860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:02.559520 6860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:02.559574 6860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.559615 6860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:02.559641 6860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.572292 6860 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:41931
I20260929 03:27:02.572352 6989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:41931 every 8 connection(s)
I20260929 03:27:02.572666 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:02.580087 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 6860
I20260929 03:27:02.580196 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:02.580372 6610 external_mini_cluster.cc:1428] Setting key 6e3a6644617196581261ceb765a42fa5
I20260929 03:27:02.581974 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.573303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:47279 (local address 127.6.116.190:42785)
0929 03:27:02.573392 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.573404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.573667 (+ 263us) server_negotiation.cc:407] Connection header received
0929 03:27:02.574108 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.574109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.574133 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.574189 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.575398 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.575870 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.576793 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.576964 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.577013 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.578995 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.578997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.578997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.579004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.580634 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.581056 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.581057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.581057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.581119 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.581409 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.581410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.581410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.581506 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.581554 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.581857 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.581913 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:27:02.582345 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.573278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:42785 (local address 127.6.116.130:47279)
0929 03:27:02.573550 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:02.573584 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:02.574038 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:02.574258 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:02.574261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:02.575137 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:02.575323 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.575327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.575962 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.575963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.576724 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.576726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.577031 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.577032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.577125 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.577564 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:02.577572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:02.578913 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:02.580718 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.580724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.580726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.580986 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.581189 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.581190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.581191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.581346 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.581568 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:02.581573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:02.581732 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 03:27:02.581839 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:27:02.582001 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":201,"threads_started":1}
I20260929 03:27:02.582778 6990 heartbeater.cc:347] Connected to a master server at 127.6.116.190:42785
I20260929 03:27:02.582878 6990 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:02.583096 6990 heartbeater.cc:511] Master 127.6.116.190:42785 requested a full tablet report, sending...
I20260929 03:27:02.583638 6662 ts_manager.cc:194] Registered new tserver with Master: 91bacae741254bf7be9cb0f64d95bc84 (127.6.116.130:41931)
I20260929 03:27:02.584012 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:47279
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:02.605777 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:02.684566 6995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:02.684824 6995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:02.684875 6995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:02.686146 6995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:02.686199 6995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:02.686237 6995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:02.687645 6995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:42785
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:02.687822 6995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:02.687988 6995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:02.692749 7003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.692749 7001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:02.692760 7000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:02.692950 6995 server_base.cc:1034] running on GCE node
I20260929 03:27:02.693250 6995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:02.693490 6995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:02.694664 6995 hybrid_clock.cc:648] HybridClock initialized: now 1790652422694650 us; error 27 us; skew 500 ppm
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:02.697154 6995 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:02.697549 6995 webserver.cc:460] Webserver started at http://127.6.116.131:39449/ using document root <none> and password file <none>
I20260929 03:27:02.697777 6995 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:02.697842 6995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:02.697989 6995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:02.698988 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "797d55f7e29e45958169bbf23e11defa"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "bf0ffe67c382f876095ee0c309fa2a23"
server_key_iv: "f3cb5b77403685a277948029aaa2ab1c"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.699406 6995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "797d55f7e29e45958169bbf23e11defa"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "bf0ffe67c382f876095ee0c309fa2a23"
server_key_iv: "f3cb5b77403685a277948029aaa2ab1c"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.700769 6995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:02.701509 7010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.701674 6995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:02.701767 6995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "797d55f7e29e45958169bbf23e11defa"
format_stamp: "Formatted at 2026-09-29 03:27:02 on dist-test-slave-z2zg"
server_key: "bf0ffe67c382f876095ee0c309fa2a23"
server_key_iv: "f3cb5b77403685a277948029aaa2ab1c"
server_key_version: "encryptionkey@0"
I20260929 03:27:02.701844 6995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:02.721364 6995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:02.723093 6995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:02.723254 6995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:02.723467 6995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:02.723762 6995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:02.723815 6995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.723856 6995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:02.723883 6995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:02.729671 6995 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:46175
I20260929 03:27:02.729756 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:46175 every 8 connection(s)
I20260929 03:27:02.730065 6995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:02.730291 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 6995
I20260929 03:27:02.730368 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:02.730548 6610 external_mini_cluster.cc:1428] Setting key 9525d44de9a8d25c2374cae923d00009
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:02.738657 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.731052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39817 (local address 127.6.116.190:42785)
0929 03:27:02.731098 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.731101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.731132 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:02.731747 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.731748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.731759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.731824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.732661 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.733142 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.734046 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.734229 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.734276 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.736236 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.736238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.736238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.736244 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.737595 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.737911 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.737911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.737912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.737958 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.738157 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.738158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.738158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.738277 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.738326 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.738541 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.738595 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:02.738853 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.730776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:42785 (local address 127.6.116.131:39817)
0929 03:27:02.730936 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:02.731058 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:02.731637 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:02.731874 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:02.731878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:02.732412 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:02.732586 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.732590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.733223 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.733224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.733976 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:02.733977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.734296 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.734297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:02.734389 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.734812 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:02.734820 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:02.736145 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:02.737660 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.737665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.737666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.737851 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.738015 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:02.738015 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:02.738016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:02.738100 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:02.738331 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:02.738335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:02.738435 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 03:27:02.738512 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:27:02.738616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":96,"threads_started":1}
I20260929 03:27:02.739342 7125 heartbeater.cc:347] Connected to a master server at 127.6.116.190:42785
I20260929 03:27:02.739415 7125 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:02.739596 7125 heartbeater.cc:511] Master 127.6.116.190:42785 requested a full tablet report, sending...
I20260929 03:27:02.739970 6662 ts_manager.cc:194] Registered new tserver with Master: 797d55f7e29e45958169bbf23e11defa (127.6.116.131:46175)
I20260929 03:27:02.740334 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:39817
I20260929 03:27:02.742326 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:02.764285 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.756763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.6.116.129:40281)
0929 03:27:02.756922 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.756927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.756939 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:27:02.756967 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.756970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.756990 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.757072 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.757620 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.758334 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.759156 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.759508 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.759597 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.761682 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.761690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.761691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.761698 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.762995 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.763447 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.763450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.763450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.763509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.763722 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.763723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.763723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.763856 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.764026 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.764071 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.764149 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260929 03:27:02.764655 6767 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59536
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[6617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:02.786182 7132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:02.778915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.6.116.129:40281)
0929 03:27:02.778999 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:02.779001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:02.779125 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:27:02.779208 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:02.779209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:02.779224 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:02.779290 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:02.779891 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.780369 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.781106 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:02.781314 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:02.781392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:02.783586 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:02.783587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:02.783588 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:02.783595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:02.785126 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.785594 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.785595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.785596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.785645 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:02.785793 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:02.785794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:02.785794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:02.785870 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:02.785952 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:02.786013 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:27:02.786079 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 03:27:02.786690 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 6724
I20260929 03:27:02.791842 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 6860
I20260929 03:27:02.796762 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 6995
I20260929 03:27:02.800747 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 6630
2026-09-29T03:27:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2243 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[7139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[7139](info): set up 2 sockets
Sep 29 03:27:02 dist-test-slave-z2zg krb5kdc[7139](info): commencing operation
krb5kdc: starting...
W20260929 03:27:03.828265 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:03 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:03.937822 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:46287
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:46287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:04.015735 7152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:04.015954 7152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:04.015990 7152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:04.017069 7152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:04.017130 7152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:04.017153 7152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:04.017182 7152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:04.017210 7152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:04.018530 7152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:46287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:46287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:04.018780 7152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:04.018982 7152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:04.023519 7158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.023500 7157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.023494 7160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.023736 7152 server_base.cc:1034] running on GCE node
I20260929 03:27:04.023965 7152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:04.024262 7152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:04.025441 7152 hybrid_clock.cc:648] HybridClock initialized: now 1790652424025444 us; error 44 us; skew 500 ppm
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:04.028059 7152 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:04.028415 7152 webserver.cc:460] Webserver started at http://127.6.116.190:36679/ using document root <none> and password file <none>
I20260929 03:27:04.028601 7152 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:04.028641 7152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:04.028725 7152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:04.029652 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "b6fcb92054a84caab067a960e0fd2093"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "df13900398a8d7c90c1cd4ba90d8f2af"
server_key_iv: "9229238d3458bc5c4fa462fd4f2c73bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.029942 7152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "b6fcb92054a84caab067a960e0fd2093"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "df13900398a8d7c90c1cd4ba90d8f2af"
server_key_iv: "9229238d3458bc5c4fa462fd4f2c73bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.031213 7152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:04.031997 7167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.032179 7152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:04.032261 7152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "b6fcb92054a84caab067a960e0fd2093"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "df13900398a8d7c90c1cd4ba90d8f2af"
server_key_iv: "9229238d3458bc5c4fa462fd4f2c73bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.032315 7152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:04.040388 7152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:04.042331 7152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:04.042433 7152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:04.045981 7152 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:46287
I20260929 03:27:04.046026 7219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:46287 every 8 connection(s)
I20260929 03:27:04.046381 7152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:04.047292 7220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:04.049365 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093: Bootstrap starting.
I20260929 03:27:04.049861 7220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:04.050091 7220 log.cc:824] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:04.050686 7220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093: No bootstrap required, opened a new log
I20260929 03:27:04.051862 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7152
I20260929 03:27:04.051970 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:04.052132 6610 external_mini_cluster.cc:1428] Setting key f539ba29b282fde32636fe90baf2d885
I20260929 03:27:04.052495 7220 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } }
I20260929 03:27:04.052676 7220 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:04.052716 7220 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fcb92054a84caab067a960e0fd2093, State: Initialized, Role: FOLLOWER
I20260929 03:27:04.052879 7220 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } }
I20260929 03:27:04.052970 7220 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:04.053000 7220 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:04.053048 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:04.053776 7220 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } }
I20260929 03:27:04.053880 7220 leader_election.cc:302] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fcb92054a84caab067a960e0fd2093; no voters:
I20260929 03:27:04.054034 7220 leader_election.cc:288] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:04.054111 7226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:04.054245 7226 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [term 1 LEADER]: Becoming Leader. State: Replica: b6fcb92054a84caab067a960e0fd2093, State: Running, Role: LEADER
I20260929 03:27:04.054277 7220 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:04.054337 7226 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } }
I20260929 03:27:04.054646 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fcb92054a84caab067a960e0fd2093. Latest consensus state: current_term: 1 leader_uuid: "b6fcb92054a84caab067a960e0fd2093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } } }
I20260929 03:27:04.054744 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:04.054656 7228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fcb92054a84caab067a960e0fd2093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fcb92054a84caab067a960e0fd2093" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46287 } } }
I20260929 03:27:04.054883 7228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:04.055012 7233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:04.055557 7233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:04.058225 7233 catalog_manager.cc:1347] Generated new cluster ID: c34c44067a5a4508ad6c617d682cb7bc
I20260929 03:27:04.058277 7233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:04.063019 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.052639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.6.116.190:46287)
0929 03:27:04.052874 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.052882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.052916 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:27:04.053040 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.053047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.053112 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.053262 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.053808 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.055328 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.056604 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.057030 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.057108 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.059809 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.059813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.059814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.059825 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.061523 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.061903 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.061907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.061907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.062003 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.062224 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.062224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.062225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.062444 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.062636 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.062702 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.062841 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
I20260929 03:27:04.070685 7233 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:04.071300 7233 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:04.078495 7233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fcb92054a84caab067a960e0fd2093: Generated new TSK 0
I20260929 03:27:04.078701 7233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:04.109866 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:04.208755 7248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:04.208951 7248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:04.208974 7248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:04.210134 7248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:04.210173 7248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:04.210198 7248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:04.211707 7248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:04.211952 7248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:04.212178 7248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:04.216920 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.216912 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.216912 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.217293 7248 server_base.cc:1034] running on GCE node
I20260929 03:27:04.217510 7248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:04.217754 7248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:04.219007 7248 hybrid_clock.cc:648] HybridClock initialized: now 1790652424218991 us; error 26 us; skew 500 ppm
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:04.221485 7248 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:04.221957 7248 webserver.cc:460] Webserver started at http://127.6.116.129:36883/ using document root <none> and password file <none>
I20260929 03:27:04.222180 7248 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:04.222246 7248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:04.222380 7248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:04.223483 7248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "fb942992b5274389ab7eb446479210ee"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "3daceb138349d7e0597a09bb3691493a"
server_key_iv: "a272260fa9ec72e9923b7a8f774d9c06"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.223850 7248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "fb942992b5274389ab7eb446479210ee"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "3daceb138349d7e0597a09bb3691493a"
server_key_iv: "a272260fa9ec72e9923b7a8f774d9c06"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.225211 7248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:04.225870 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.226037 7248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:04.226133 7248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "fb942992b5274389ab7eb446479210ee"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "3daceb138349d7e0597a09bb3691493a"
server_key_iv: "a272260fa9ec72e9923b7a8f774d9c06"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.226207 7248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:04.244685 7248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:04.246451 7248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:04.246618 7248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:04.246841 7248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:04.247155 7248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:04.247211 7248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.247249 7248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:04.247278 7248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.252812 7248 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:45997
I20260929 03:27:04.252863 7377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:45997 every 8 connection(s)
I20260929 03:27:04.253161 7248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:04.255155 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7248
I20260929 03:27:04.255357 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:04.255601 6610 external_mini_cluster.cc:1428] Setting key 1786c139a963fdca735023911cbb6310
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:04.263264 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.253806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:59201 (local address 127.6.116.190:46287)
0929 03:27:04.253902 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.253904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.254092 (+ 188us) server_negotiation.cc:407] Connection header received
0929 03:27:04.255077 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.255078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.255091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.255167 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.255945 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.256601 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.257613 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.257819 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.257902 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.260405 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.260406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.260406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.260415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.261796 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.262269 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.262270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.262270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.262321 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.262602 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.262602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.262603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.262702 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.262751 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.263121 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.263201 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:27:04.263523 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.253778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46287 (local address 127.6.116.129:59201)
0929 03:27:04.253960 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.254003 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.254931 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.255245 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.255249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.255594 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:04.255853 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.255859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.256700 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.256701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.257506 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.257508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.257914 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.257915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.258066 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.258584 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:04.258592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:04.260294 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:04.261878 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.261885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.261887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.262190 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.262391 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.262392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.262392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.262535 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.262769 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:04.262775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:04.262939 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.263092 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.263270 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:04.264078 7378 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46287
I20260929 03:27:04.264190 7378 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:04.264357 7378 heartbeater.cc:511] Master 127.6.116.190:46287 requested a full tablet report, sending...
I20260929 03:27:04.264807 7184 ts_manager.cc:194] Registered new tserver with Master: fb942992b5274389ab7eb446479210ee (127.6.116.129:45997)
I20260929 03:27:04.265399 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:59201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:04.281519 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:04.362474 7383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:04.362682 7383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:04.362720 7383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:04.363931 7383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:04.363997 7383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:04.364041 7383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:04.365483 7383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:04.365736 7383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:04.365948 7383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:04.370543 7388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.370556 7391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.370797 7383 server_base.cc:1034] running on GCE node
W20260929 03:27:04.370553 7389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.371031 7383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:04.371315 7383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:04.372452 7383 hybrid_clock.cc:648] HybridClock initialized: now 1790652424372443 us; error 26 us; skew 500 ppm
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:04.374882 7383 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:04.375411 7383 webserver.cc:460] Webserver started at http://127.6.116.130:38925/ using document root <none> and password file <none>
I20260929 03:27:04.375645 7383 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:04.375715 7383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:04.375844 7383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:04.376824 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "295a48f2e1b24ac3b097308ce7c3ca2e"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "7e05b3f16c43430715fd3bd4402b0e8e"
server_key_iv: "e23bc22271c7d4651e62cf39c32e3ef7"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.377182 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "295a48f2e1b24ac3b097308ce7c3ca2e"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "7e05b3f16c43430715fd3bd4402b0e8e"
server_key_iv: "e23bc22271c7d4651e62cf39c32e3ef7"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.378443 7383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:04.379231 7398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.379415 7383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:04.379513 7383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "295a48f2e1b24ac3b097308ce7c3ca2e"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "7e05b3f16c43430715fd3bd4402b0e8e"
server_key_iv: "e23bc22271c7d4651e62cf39c32e3ef7"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.379624 7383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:04.399966 7383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:04.401815 7383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:04.401973 7383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:04.402185 7383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:04.402467 7383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:04.402520 7383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.402570 7383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:04.402617 7383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.408272 7383 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:41993
I20260929 03:27:04.408314 7513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:41993 every 8 connection(s)
I20260929 03:27:04.408622 7383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:04.415769 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7383
I20260929 03:27:04.415840 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:04.415998 6610 external_mini_cluster.cc:1428] Setting key 542f99db4669692d3fd711fe6a0124a4
I20260929 03:27:04.417589 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.409244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:52063 (local address 127.6.116.190:46287)
0929 03:27:04.409310 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.409313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.409513 (+ 200us) server_negotiation.cc:407] Connection header received
0929 03:27:04.410336 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.410337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.410346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.410392 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.410920 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.411481 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.412418 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.412603 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.412657 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.414710 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.414711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.414711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.414718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.416162 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.416578 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.416580 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.416580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.416646 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.416965 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.416966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.416967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.417075 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.417130 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.417442 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.417506 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:27:04.417868 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.409222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46287 (local address 127.6.116.130:52063)
0929 03:27:04.409399 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.409430 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.410253 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.410447 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.410450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.410685 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:04.410850 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.410854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.411554 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.411555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.412345 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.412347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.412676 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.412677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.412770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.413302 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:04.413309 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:04.414627 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:04.416228 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.416235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.416237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.416458 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.416732 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.416733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.416733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.416879 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.417144 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:04.417150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:04.417311 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.417422 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.417590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260929 03:27:04.418380 7514 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46287
I20260929 03:27:04.418486 7514 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:04.418649 7514 heartbeater.cc:511] Master 127.6.116.190:46287 requested a full tablet report, sending...
I20260929 03:27:04.419020 7184 ts_manager.cc:194] Registered new tserver with Master: 295a48f2e1b24ac3b097308ce7c3ca2e (127.6.116.130:41993)
I20260929 03:27:04.419529 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:52063
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:04.441340 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:04.518925 7519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:04.519145 7519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:04.519191 7519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:04.520365 7519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:04.520426 7519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:04.520469 7519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:04.521925 7519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46287
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:04.522156 7519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:04.522394 7519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:04.527127 7525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.527216 7519 server_base.cc:1034] running on GCE node
W20260929 03:27:04.527132 7524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:04.527132 7527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:04.527506 7519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:04.527760 7519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:04.528914 7519 hybrid_clock.cc:648] HybridClock initialized: now 1790652424528893 us; error 50 us; skew 500 ppm
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:04.531351 7519 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:04.531800 7519 webserver.cc:460] Webserver started at http://127.6.116.131:40203/ using document root <none> and password file <none>
I20260929 03:27:04.532059 7519 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:04.532140 7519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:04.532280 7519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:04.533258 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "8189a63d2b8c430fbbba332c810e62e9"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "a593bb8822c4ddd17442b7f4ee7ba27a"
server_key_iv: "802b8ce9b6c988054b003039da10cb5f"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.533607 7519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "8189a63d2b8c430fbbba332c810e62e9"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "a593bb8822c4ddd17442b7f4ee7ba27a"
server_key_iv: "802b8ce9b6c988054b003039da10cb5f"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.534861 7519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:04.535665 7534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.535832 7519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.535926 7519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "8189a63d2b8c430fbbba332c810e62e9"
format_stamp: "Formatted at 2026-09-29 03:27:04 on dist-test-slave-z2zg"
server_key: "a593bb8822c4ddd17442b7f4ee7ba27a"
server_key_iv: "802b8ce9b6c988054b003039da10cb5f"
server_key_version: "encryptionkey@0"
I20260929 03:27:04.536012 7519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:04.544709 7519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:04.546713 7519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:04.546864 7519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:04.547122 7519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:04.547407 7519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:04.547461 7519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.547499 7519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:04.547526 7519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:04.553134 7519 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:39077
I20260929 03:27:04.553179 7648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:39077 every 8 connection(s)
I20260929 03:27:04.553462 7519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:04.555339 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7519
I20260929 03:27:04.555457 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:04.555639 6610 external_mini_cluster.cc:1428] Setting key 8fb991a208eef7fb5e689ddec4518850
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:04.565053 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.554145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51135 (local address 127.6.116.190:46287)
0929 03:27:04.554186 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.554188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.554394 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:27:04.555623 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.555625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.555636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.555678 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.556541 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.557276 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.558684 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.558957 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.559024 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.562344 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.562345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.562366 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.562374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.563971 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.564329 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.564330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.564330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.564379 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.564574 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.564575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.564575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.564679 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.564726 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.564935 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.564991 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:04.565201 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.554123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46287 (local address 127.6.116.131:51135)
0929 03:27:04.554262 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.554292 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.555438 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.555893 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.555897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.556213 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:04.556447 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.556452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.557378 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.557379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.558590 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.558592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.559064 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.559066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.559205 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.559864 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:04.559873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:04.562091 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:04.564035 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.564040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.564042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.564264 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.564435 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:04.564435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:04.564435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:04.564523 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:04.564731 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:04.564735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:04.564832 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.564908 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.565013 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260929 03:27:04.565639 7649 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46287
I20260929 03:27:04.565713 7649 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:04.565909 7649 heartbeater.cc:511] Master 127.6.116.190:46287 requested a full tablet report, sending...
I20260929 03:27:04.566329 7184 ts_manager.cc:194] Registered new tserver with Master: 8189a63d2b8c430fbbba332c810e62e9 (127.6.116.131:39077)
I20260929 03:27:04.566654 7184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:51135
I20260929 03:27:04.574455 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:27:04.581362 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.575943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.6.116.190:46287)
0929 03:27:04.576007 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.576009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.576113 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:27:04.576191 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.576192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.576201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.576248 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.576771 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.577238 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.577937 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.578139 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.578226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.579463 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.579465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.579465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.579471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.580548 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.580825 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.580825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.580826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.580870 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.581006 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.581006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.581006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.581114 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.581186 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.581259 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.581309 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:04.583330 7184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:04.584843 7184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:04.590916 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.587160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.6.116.129:45997)
0929 03:27:04.587316 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.587321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.587476 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:27:04.587557 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.587560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.587576 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.587659 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:04.587855 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.588955 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.589962 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.590490 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.590557 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.590608 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.590823 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.590860 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260929 03:27:04.590921 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.586895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:45997 (local address 127.0.0.1:47154)
0929 03:27:04.587403 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.587412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.587485 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.587681 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.587683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.587685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:04.587756 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.587757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.589075 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.589076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.589871 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.589873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.590642 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.590643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.590721 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.590732 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.590807 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.590851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
I20260929 03:27:04.591619 7311 tablet_service.cc:1467] Processing CreateTablet for tablet dffbb3f26e334f9d8ac4a4505a555f2c (DEFAULT_TABLE table=test-table [id=a118bf6adf984342991168cbea1243fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:04.591956 7311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffbb3f26e334f9d8ac4a4505a555f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:04.592198 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.586895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39077 (local address 127.0.0.1:55540)
0929 03:27:04.587029 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.587088 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.587152 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.587749 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.587750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.587752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:04.587788 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.587790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.589090 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.589091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.590359 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.590362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.591966 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.591967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.592032 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.592036 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.592095 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.592135 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":78,"thread_start_us":46,"threads_started":1}
I20260929 03:27:04.592422 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.586982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55540 (local address 127.6.116.131:39077)
0929 03:27:04.587607 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.587611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.587619 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:04.587649 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.587651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.587663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.587738 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:04.587872 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.588958 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.591390 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.591891 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.591948 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.591979 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.592303 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.592346 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":101,"threads_started":1}
I20260929 03:27:04.593044 7582 tablet_service.cc:1467] Processing CreateTablet for tablet dffbb3f26e334f9d8ac4a4505a555f2c (DEFAULT_TABLE table=test-table [id=a118bf6adf984342991168cbea1243fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:04.593369 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffbb3f26e334f9d8ac4a4505a555f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:04.594780 7666 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Bootstrap starting.
I20260929 03:27:04.595104 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.587414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.6.116.130:41993)
0929 03:27:04.587556 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.587560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.587951 (+ 391us) server_negotiation.cc:407] Connection header received
0929 03:27:04.587991 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.587994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.588010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.588079 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:04.588287 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.588955 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.590449 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.591230 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.591326 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.591362 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.594931 (+ 3569us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.594980 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260929 03:27:04.595357 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.587106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:41993 (local address 127.0.0.1:55328)
0929 03:27:04.587840 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.587850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.587896 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.588161 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.588162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.588163 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:04.588206 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.588207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.589056 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.589057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.590361 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.590362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.594766 (+ 4404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.594767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.594834 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.594838 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.595243 (+ 405us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.595288 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":64,"thread_start_us":83,"threads_started":1}
I20260929 03:27:04.595463 7666 tablet_bootstrap.cc:654] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:04.596020 7667 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Bootstrap starting.
I20260929 03:27:04.595990 7446 tablet_service.cc:1467] Processing CreateTablet for tablet dffbb3f26e334f9d8ac4a4505a555f2c (DEFAULT_TABLE table=test-table [id=a118bf6adf984342991168cbea1243fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:04.596279 7666 log.cc:824] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:04.596316 7446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dffbb3f26e334f9d8ac4a4505a555f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:04.596724 7667 tablet_bootstrap.cc:654] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:04.596998 7666 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: No bootstrap required, opened a new log
I20260929 03:27:04.597080 7666 ts_tablet_manager.cc:1397] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:27:04.597267 7667 log.cc:824] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:04.598016 7667 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: No bootstrap required, opened a new log
I20260929 03:27:04.598086 7667 ts_tablet_manager.cc:1397] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:04.598677 7666 raft_consensus.cc:348] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.598832 7666 raft_consensus.cc:374] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:04.598875 7666 raft_consensus.cc:729] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb942992b5274389ab7eb446479210ee, State: Initialized, Role: FOLLOWER
I20260929 03:27:04.599015 7666 consensus_queue.cc:260] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.599509 7666 ts_tablet_manager.cc:1428] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:04.599514 7670 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Bootstrap starting.
I20260929 03:27:04.599635 7378 heartbeater.cc:503] Master 127.6.116.190:46287 was elected leader, sending a full tablet report...
I20260929 03:27:04.600222 7670 tablet_bootstrap.cc:654] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:04.600235 7667 raft_consensus.cc:348] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.600414 7667 raft_consensus.cc:374] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:04.600453 7667 raft_consensus.cc:729] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8189a63d2b8c430fbbba332c810e62e9, State: Initialized, Role: FOLLOWER
I20260929 03:27:04.600564 7667 consensus_queue.cc:260] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.600806 7670 log.cc:824] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:04.600833 7667 ts_tablet_manager.cc:1428] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:04.601019 7649 heartbeater.cc:503] Master 127.6.116.190:46287 was elected leader, sending a full tablet report...
I20260929 03:27:04.601461 7670 tablet_bootstrap.cc:492] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: No bootstrap required, opened a new log
I20260929 03:27:04.601545 7670 ts_tablet_manager.cc:1397] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:04.602797 7670 raft_consensus.cc:348] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.602947 7670 raft_consensus.cc:374] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:04.602990 7670 raft_consensus.cc:729] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295a48f2e1b24ac3b097308ce7c3ca2e, State: Initialized, Role: FOLLOWER
I20260929 03:27:04.603111 7670 consensus_queue.cc:260] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.603387 7670 ts_tablet_manager.cc:1428] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:04.603407 7514 heartbeater.cc:503] Master 127.6.116.190:46287 was elected leader, sending a full tablet report...
W20260929 03:27:04.659209 7515 tablet.cc:2369] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:04.743990 7674 raft_consensus.cc:482] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:04.744105 7674 raft_consensus.cc:504] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.744427 7674 leader_election.cc:288] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8189a63d2b8c430fbbba332c810e62e9 (127.6.116.131:39077), fb942992b5274389ab7eb446479210ee (127.6.116.129:45997)
I20260929 03:27:04.747318 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.744699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:50191 (local address 127.6.116.131:39077)
0929 03:27:04.744743 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.744745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.744772 (+ 27us) server_negotiation.cc:407] Connection header received
0929 03:27:04.744800 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.744801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.744813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.744871 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:04.745047 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.745553 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.746560 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.746927 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.746982 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.747013 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.747224 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.747255 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:04.747313 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.744693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:57749 (local address 127.6.116.129:45997)
0929 03:27:04.744737 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.744739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.744877 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:27:04.744907 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.744908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.744919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.744982 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:04.745154 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.745624 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.746560 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.746933 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.747022 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.747029 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.747196 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.747235 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:04.747314 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.744607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39077 (local address 127.6.116.130:50191)
0929 03:27:04.744662 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.744674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.744718 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.744892 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.744893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.744895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:04.744971 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.744973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.745634 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.745635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.746458 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.746461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.747010 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.747011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.747124 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.747127 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.747205 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.747242 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":29}
I20260929 03:27:04.747326 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.744617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:45997 (local address 127.6.116.130:57749)
0929 03:27:04.744796 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:04.744806 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:04.744864 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:04.744994 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:04.744995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:04.744996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:04.745081 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.745082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.745706 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.745707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.746466 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:04.746467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.747066 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.747067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:04.747118 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.747122 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:04.747172 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:27:04.747235 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":12,"thread_start_us":88,"threads_started":1}
I20260929 03:27:04.747651 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffbb3f26e334f9d8ac4a4505a555f2c" candidate_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8189a63d2b8c430fbbba332c810e62e9" is_pre_election: true
I20260929 03:27:04.747813 7603 raft_consensus.cc:2454] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295a48f2e1b24ac3b097308ce7c3ca2e in term 0.
I20260929 03:27:04.747800 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffbb3f26e334f9d8ac4a4505a555f2c" candidate_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb942992b5274389ab7eb446479210ee" is_pre_election: true
I20260929 03:27:04.747982 7331 raft_consensus.cc:2454] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 295a48f2e1b24ac3b097308ce7c3ca2e in term 0.
I20260929 03:27:04.748051 7399 leader_election.cc:302] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295a48f2e1b24ac3b097308ce7c3ca2e, 8189a63d2b8c430fbbba332c810e62e9; no voters:
I20260929 03:27:04.748191 7674 raft_consensus.cc:2781] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:04.748279 7674 raft_consensus.cc:482] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:04.748315 7674 raft_consensus.cc:3037] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:04.748982 7674 raft_consensus.cc:504] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.749127 7674 leader_election.cc:288] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [CANDIDATE]: Term 1 election: Requested vote from peers 8189a63d2b8c430fbbba332c810e62e9 (127.6.116.131:39077), fb942992b5274389ab7eb446479210ee (127.6.116.129:45997)
I20260929 03:27:04.749264 7603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffbb3f26e334f9d8ac4a4505a555f2c" candidate_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8189a63d2b8c430fbbba332c810e62e9"
I20260929 03:27:04.749366 7603 raft_consensus.cc:3037] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:04.749342 7331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dffbb3f26e334f9d8ac4a4505a555f2c" candidate_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb942992b5274389ab7eb446479210ee"
I20260929 03:27:04.749436 7331 raft_consensus.cc:3037] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:04.750134 7603 raft_consensus.cc:2454] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295a48f2e1b24ac3b097308ce7c3ca2e in term 1.
I20260929 03:27:04.750188 7331 raft_consensus.cc:2454] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 295a48f2e1b24ac3b097308ce7c3ca2e in term 1.
I20260929 03:27:04.750325 7399 leader_election.cc:302] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 295a48f2e1b24ac3b097308ce7c3ca2e, 8189a63d2b8c430fbbba332c810e62e9; no voters:
I20260929 03:27:04.750406 7674 raft_consensus.cc:2781] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:04.750572 7674 raft_consensus.cc:686] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [term 1 LEADER]: Becoming Leader. State: Replica: 295a48f2e1b24ac3b097308ce7c3ca2e, State: Running, Role: LEADER
I20260929 03:27:04.750682 7674 consensus_queue.cc:237] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } }
I20260929 03:27:04.751495 7184 catalog_manager.cc:5495] T dffbb3f26e334f9d8ac4a4505a555f2c P 295a48f2e1b24ac3b097308ce7c3ca2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 295a48f2e1b24ac3b097308ce7c3ca2e (127.6.116.130). New cstate: current_term: 1 leader_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189a63d2b8c430fbbba332c810e62e9" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb942992b5274389ab7eb446479210ee" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 45997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "295a48f2e1b24ac3b097308ce7c3ca2e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41993 } health_report { overall_health: HEALTHY } } }
W20260929 03:27:04.753799 7379 tablet.cc:2369] T dffbb3f26e334f9d8ac4a4505a555f2c P fb942992b5274389ab7eb446479210ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:04.775787 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.768343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.6.116.129:45997)
0929 03:27:04.768433 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.768435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.768536 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:27:04.768617 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.768618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.768629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.768711 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.769265 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.769709 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.770411 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.770593 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.770682 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.773055 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.773065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.773065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.773071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.774569 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.774969 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.774972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.774972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.775079 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.775284 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.775285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.775285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.775426 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.775536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.775588 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.775655 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:04.798071 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.791291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.6.116.129:45997)
0929 03:27:04.791362 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.791365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.791453 (+ 88us) server_negotiation.cc:407] Connection header received
0929 03:27:04.791548 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.791549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.791565 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.791622 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.792189 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.792636 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.793355 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.793541 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.793621 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.795777 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.795778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.795778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.795786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.796912 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.797262 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.797264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.797264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.797320 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.797519 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.797520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.797521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.797622 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.797724 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.797893 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.797970 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 03:27:04.803938 7650 tablet.cc:2369] T dffbb3f26e334f9d8ac4a4505a555f2c P 8189a63d2b8c430fbbba332c810e62e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7139](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:04.818902 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:04.812255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.6.116.129:45997)
0929 03:27:04.812339 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:04.812341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:04.812437 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:27:04.812517 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:04.812518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:04.812535 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:04.812610 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:04.813185 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.813639 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.814373 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:04.814560 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:04.814645 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:04.816728 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:04.816729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:04.816729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:04.816744 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:04.817841 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.818184 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.818185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.818186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.818238 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:04.818432 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:04.818433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:04.818433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:04.818527 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:04.818628 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:04.818731 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:27:04.818802 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 03:27:04.819368 7291 tablet_service.cc:2908] Rejecting scan request for tablet dffbb3f26e334f9d8ac4a4505a555f2c: Uninitialized: clean time has not yet been initialized
I20260929 03:27:04.820132 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7248
I20260929 03:27:04.825647 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7383
I20260929 03:27:04.830106 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7519
I20260929 03:27:04.834120 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7152
2026-09-29T03:27:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2034 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7690](info): set up 2 sockets
Sep 29 03:27:04 dist-test-slave-z2zg krb5kdc[7690](info): commencing operation
krb5kdc: starting...
W20260929 03:27:05.856441 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:05 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:05.970413 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45013
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:45013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:06.049314 7703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:06.049525 7703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:06.049551 7703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:06.050613 7703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:06.050650 7703 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:27:06.050666 7703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:06.050681 7703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:06.050695 7703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:06.050710 7703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:06.052106 7703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:45013
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:06.052297 7703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:06.052464 7703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:06.057116 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.057133 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.057137 7703 server_base.cc:1034] running on GCE node
W20260929 03:27:06.057117 7711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.057549 7703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:06.057835 7703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:06.059094 7703 hybrid_clock.cc:648] HybridClock initialized: now 1790652426059032 us; error 48 us; skew 500 ppm
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:06.066095 7703 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:06.066521 7703 webserver.cc:460] Webserver started at http://127.6.116.190:33165/ using document root <none> and password file <none>
I20260929 03:27:06.066746 7703 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:06.066814 7703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:06.066956 7703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:06.068010 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "16d74335631d4776ab011df266cba19e"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "5c2998b2dea87b9571a12bfcd2fe47b3"
server_key_iv: "e9a798e09e02a8bc9113e35c39d13082"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.068372 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "16d74335631d4776ab011df266cba19e"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "5c2998b2dea87b9571a12bfcd2fe47b3"
server_key_iv: "e9a798e09e02a8bc9113e35c39d13082"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.069686 7703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:06.070477 7718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.070938 7703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:06.071089 7703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "16d74335631d4776ab011df266cba19e"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "5c2998b2dea87b9571a12bfcd2fe47b3"
server_key_iv: "e9a798e09e02a8bc9113e35c39d13082"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.071179 7703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:06.105611 7703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:06.107838 7703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:06.108011 7703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:06.112771 7703 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:45013
I20260929 03:27:06.112820 7781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:45013 every 8 connection(s)
I20260929 03:27:06.113147 7703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:06.113744 7782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.114776 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7703
I20260929 03:27:06.114847 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:06.115085 6610 external_mini_cluster.cc:1428] Setting key 7603b298f48251bf5b8b01d6f8d46d99
I20260929 03:27:06.116577 7782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Bootstrap starting.
I20260929 03:27:06.117179 7782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.117493 7782 log.cc:824] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:06.118216 7782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: No bootstrap required, opened a new log
I20260929 03:27:06.120559 7782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } }
I20260929 03:27:06.120744 7782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.120774 7782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d74335631d4776ab011df266cba19e, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.120911 7782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } }
I20260929 03:27:06.120990 7782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:06.121011 7782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:06.121062 7782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.121549 7782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } }
I20260929 03:27:06.121627 7782 leader_election.cc:302] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16d74335631d4776ab011df266cba19e; no voters:
I20260929 03:27:06.121800 7782 leader_election.cc:288] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:06.121832 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:06.121949 7787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [term 1 LEADER]: Becoming Leader. State: Replica: 16d74335631d4776ab011df266cba19e, State: Running, Role: LEADER
I20260929 03:27:06.122042 7782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:06.122031 7787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } }
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.122376 7787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16d74335631d4776ab011df266cba19e. Latest consensus state: current_term: 1 leader_uuid: "16d74335631d4776ab011df266cba19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } } }
I20260929 03:27:06.122470 7787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:06.122392 7789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16d74335631d4776ab011df266cba19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16d74335631d4776ab011df266cba19e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45013 } } }
I20260929 03:27:06.122836 7789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:06.123368 7794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:06.123842 7794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:06.125337 7794 catalog_manager.cc:1347] Generated new cluster ID: 710d67b6307f4525804000a266a90a60
I20260929 03:27:06.125388 7794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:06.126888 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.115638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.6.116.190:45013)
0929 03:27:06.115832 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.115841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.115874 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:27:06.116374 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.116382 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.116441 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.116592 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.117227 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.118524 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.119404 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.120051 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.120162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.122863 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.122867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.122868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.122879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.125053 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.125458 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.125461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.125461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.125559 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.125752 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.125753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.125753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.126009 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.126130 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.126546 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.126678 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260929 03:27:06.133749 7794 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:06.134243 7794 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:06.137290 7794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Generated new TSK 0
I20260929 03:27:06.137485 7794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:06.162492 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45013
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:06.250924 7808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:06.251184 7808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:06.251226 7808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:06.252333 7808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:06.252396 7808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:06.252432 7808 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:06.252468 7808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:06.253975 7808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:45013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:06.254231 7808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:06.254441 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:06.260111 7814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.260140 7813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.260092 7816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.260408 7808 server_base.cc:1034] running on GCE node
I20260929 03:27:06.260637 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:06.260905 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:06.262059 7808 hybrid_clock.cc:648] HybridClock initialized: now 1790652426262043 us; error 31 us; skew 500 ppm
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:06.264525 7808 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:06.264914 7808 webserver.cc:460] Webserver started at http://127.6.116.129:45303/ using document root <none> and password file <none>
I20260929 03:27:06.265131 7808 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:06.265205 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:06.265352 7808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:06.266322 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "995d486e47c3493e824d276e0347c863"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "41d9538477409fb092f2d196be9c07e1"
server_key_iv: "a5d735282391bacb6c3305a3cff053ec"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.266678 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "995d486e47c3493e824d276e0347c863"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "41d9538477409fb092f2d196be9c07e1"
server_key_iv: "a5d735282391bacb6c3305a3cff053ec"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.268050 7808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:06.268812 7823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.268975 7808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.269057 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "995d486e47c3493e824d276e0347c863"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "41d9538477409fb092f2d196be9c07e1"
server_key_iv: "a5d735282391bacb6c3305a3cff053ec"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.269109 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:06.277079 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:06.278700 7808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:06.278800 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:06.279258 7808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:06.279302 7808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.279323 7808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:06.279361 7808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.285419 7808 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:33807
I20260929 03:27:06.285456 7939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:33807 every 8 connection(s)
I20260929 03:27:06.285792 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.287101 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7808
I20260929 03:27:06.287202 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:06.287369 6610 external_mini_cluster.cc:1428] Setting key 6bf379ae5d6ab59ab8d8fbbc94b62dcb
I20260929 03:27:06.289373 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.279696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:40365 (local address 127.6.116.190:45013)
0929 03:27:06.279763 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.279766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.279913 (+ 147us) server_negotiation.cc:407] Connection header received
0929 03:27:06.280338 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.280339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.280351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.280432 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.281194 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.281993 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.283582 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.283907 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.283993 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.286435 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.286436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.286436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.286442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.288055 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.288448 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.288449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.288449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.288508 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.288736 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.288737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.288737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.288873 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.288940 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.289147 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.289246 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:27:06.289381 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.279636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.129:40365)
0929 03:27:06.279799 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.279834 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.280254 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.280476 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.280479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.280832 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.281088 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.281094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.282095 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.282096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.283493 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.283496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.283996 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.283997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.284095 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.284675 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.284685 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.286356 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.288147 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.288152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.288154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.288366 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.288574 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.288575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.288575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.288677 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.288935 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.288938 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.289034 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.289110 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.289212 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260929 03:27:06.289897 7940 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45013
I20260929 03:27:06.289978 7940 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:06.290146 7940 heartbeater.cc:511] Master 127.6.116.190:45013 requested a full tablet report, sending...
I20260929 03:27:06.290638 7734 ts_manager.cc:194] Registered new tserver with Master: 995d486e47c3493e824d276e0347c863 (127.6.116.129:33807)
I20260929 03:27:06.291564 7734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:40365
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:06.296949 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.290902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:46135 (local address 127.6.116.190:45013)
0929 03:27:06.290940 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.290942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.291064 (+ 122us) server_negotiation.cc:407] Connection header received
0929 03:27:06.291125 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.291126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.291136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.291186 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.291497 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.292248 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.293345 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.293514 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.293570 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.294544 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.294545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.294545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.294551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.295950 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.296348 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.296353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.296353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.296398 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.296586 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.296587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.296587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.296670 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.296747 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.296844 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.296898 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:27:06.296957 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.290851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.129:46135)
0929 03:27:06.290979 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.290988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.291060 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.291197 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.291198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.291380 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.291430 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.291431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.292362 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.292363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.293234 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.293237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.293593 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.293594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.293689 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.294075 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.294081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.294447 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.296030 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.296031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.296032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.296267 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.296465 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.296465 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.296466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.296528 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.296742 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.296743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.296776 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.296826 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.296894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:06.313957 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45013
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:06.394640 7949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:06.394812 7949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:06.394833 7949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:06.396016 7949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:06.396054 7949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:06.396075 7949 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:06.396117 7949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:06.397506 7949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:45013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:06.397689 7949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:06.397894 7949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:06.402412 7954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.402413 7957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.402426 7955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.402668 7949 server_base.cc:1034] running on GCE node
I20260929 03:27:06.402864 7949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:06.403095 7949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:06.404230 7949 hybrid_clock.cc:648] HybridClock initialized: now 1790652426404218 us; error 38 us; skew 500 ppm
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:06.406544 7949 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:06.406931 7949 webserver.cc:460] Webserver started at http://127.6.116.130:39951/ using document root <none> and password file <none>
I20260929 03:27:06.407219 7949 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:06.407264 7949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:06.407405 7949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:06.408428 7949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "0b5dcaa00c2946d087d6e30553241261"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "1c30494e6ac6bdf47bdae5528763d5a8"
server_key_iv: "de0ae056a18f21590322ac80bf686554"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.408757 7949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "0b5dcaa00c2946d087d6e30553241261"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "1c30494e6ac6bdf47bdae5528763d5a8"
server_key_iv: "de0ae056a18f21590322ac80bf686554"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.410137 7949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:06.410742 7964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.410888 7949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:06.410970 7949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "0b5dcaa00c2946d087d6e30553241261"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "1c30494e6ac6bdf47bdae5528763d5a8"
server_key_iv: "de0ae056a18f21590322ac80bf686554"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.411046 7949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:06.426558 7949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:06.428354 7949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:06.428509 7949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:06.429131 7949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:06.429176 7949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.429209 7949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:06.429232 7949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.436040 7949 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45183
I20260929 03:27:06.436143 8080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45183 every 8 connection(s)
I20260929 03:27:06.436383 7949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:06.438194 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 7949
I20260929 03:27:06.438277 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:06.438452 6610 external_mini_cluster.cc:1428] Setting key 361a636440ec97de51f0cf78ad49ff82
I20260929 03:27:06.439014 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.429691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:50011 (local address 127.6.116.190:45013)
0929 03:27:06.429729 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.429731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.429906 (+ 175us) server_negotiation.cc:407] Connection header received
0929 03:27:06.430358 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.430359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.430373 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.430430 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.431249 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.431753 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.432746 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.432993 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.433075 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.436349 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.436350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.436350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.436358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.437644 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.437998 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.437999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.437999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.438079 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.438358 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.438359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.438359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.438473 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.438552 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.438839 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.438903 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:06.439224 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.429647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.130:50011)
0929 03:27:06.429790 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.429830 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.430251 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.430492 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.430496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.430873 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.431155 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.431161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.431854 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.431855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.432649 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.432658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.433083 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.433084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.433206 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.433691 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.433701 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.436255 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.437719 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.437723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.437725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.437934 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.438163 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.438164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.438164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.438272 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.438577 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.438582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.438694 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.438822 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.438964 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260929 03:27:06.439659 8081 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45013
I20260929 03:27:06.439747 8081 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:06.439919 8081 heartbeater.cc:511] Master 127.6.116.190:45013 requested a full tablet report, sending...
I20260929 03:27:06.440254 7735 ts_manager.cc:194] Registered new tserver with Master: 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183)
I20260929 03:27:06.440729 7735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:50011
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:27:06.446172 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.440742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:44577 (local address 127.6.116.190:45013)
0929 03:27:06.440774 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.440776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.440953 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:27:06.441022 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.441023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.441033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.441090 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.441407 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.441865 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.442562 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.442767 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.442848 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.443733 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.443734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.443735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.443741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.445070 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.445413 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.445414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.445415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.445468 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.445676 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.445677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.445677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.445810 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.445883 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.446043 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.446114 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:06.446480 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.440590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.130:44577)
0929 03:27:06.440900 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.440909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.440966 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.441109 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.441109 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.441274 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.441322 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.441323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.441954 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.441955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.442465 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.442466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.442850 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.442851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.442927 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.443366 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.443371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.443627 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.445171 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.445172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.445172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.445332 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.445537 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.445538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.445539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.445592 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.445876 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.445877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.445918 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.446324 (+ 406us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.446425 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:06.462073 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45013
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:06.540277 8090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:06.540527 8090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:06.540565 8090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:06.541654 8090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:06.541702 8090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:06.541723 8090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:06.541739 8090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:06.543028 8090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:45013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:06.543272 8090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:06.543426 8090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:06.548043 8097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.548074 8090 server_base.cc:1034] running on GCE node
W20260929 03:27:06.548043 8096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:06.548022 8099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:06.548424 8090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:06.548673 8090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:06.549850 8090 hybrid_clock.cc:648] HybridClock initialized: now 1790652426549828 us; error 32 us; skew 500 ppm
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:06.553007 8090 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:06.553445 8090 webserver.cc:460] Webserver started at http://127.6.116.131:41207/ using document root <none> and password file <none>
I20260929 03:27:06.553629 8090 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:06.553678 8090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:06.553759 8090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:06.554705 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "6d4bf86393b94541a4bfc18742e6b141"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "27735341afe668cb4622da71c20fea0f"
server_key_iv: "fe39c1322efd5d3b99c4489d50ade467"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.554988 8090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "6d4bf86393b94541a4bfc18742e6b141"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "27735341afe668cb4622da71c20fea0f"
server_key_iv: "fe39c1322efd5d3b99c4489d50ade467"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.556313 8090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:06.556916 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.557081 8090 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:06.557174 8090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "6d4bf86393b94541a4bfc18742e6b141"
format_stamp: "Formatted at 2026-09-29 03:27:06 on dist-test-slave-z2zg"
server_key: "27735341afe668cb4622da71c20fea0f"
server_key_iv: "fe39c1322efd5d3b99c4489d50ade467"
server_key_version: "encryptionkey@0"
I20260929 03:27:06.557260 8090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:06.570147 8090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:06.571976 8090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:06.572140 8090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:06.572628 8090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:06.572685 8090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.572731 8090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:06.572762 8090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.579793 8090 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:45809
I20260929 03:27:06.580098 8090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:06.580082 8221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:45809 every 8 connection(s)
I20260929 03:27:06.582226 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.572939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:38463 (local address 127.6.116.190:45013)
0929 03:27:06.573006 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.573009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.573248 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:27:06.573707 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.573709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.573721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.573781 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.574601 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.575192 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.576323 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.576537 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.576622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.579516 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.579517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.579518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.579525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.581122 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.581501 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.581502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.581502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.581552 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.581721 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.581722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.581722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.581838 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.581888 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.582098 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.582156 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:27:06.582360 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.572932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.131:38463)
0929 03:27:06.573104 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.573140 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.573588 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.573849 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.573853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.574234 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.574507 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.574513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.575309 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.575310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.576236 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.576238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.576630 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.576631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.576753 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.577233 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.577252 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.579421 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.581192 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.581195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.581197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.581437 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.581594 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.581594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.581594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.581674 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.581894 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.581898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.581995 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.582072 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.582184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260929 03:27:06.582827 8222 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45013
I20260929 03:27:06.582906 8222 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:06.583118 8222 heartbeater.cc:511] Master 127.6.116.190:45013 requested a full tablet report, sending...
I20260929 03:27:06.583525 7734 ts_manager.cc:194] Registered new tserver with Master: 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809)
I20260929 03:27:06.584018 7734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:38463
I20260929 03:27:06.586092 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 8090
I20260929 03:27:06.586200 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:06.586362 6610 external_mini_cluster.cc:1428] Setting key 0d59796b85cc42e16c08f05be825c025
I20260929 03:27:06.587150 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:27:06.589367 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.583830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.6.116.131:57735)
0929 03:27:06.583946 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.583957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.584012 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.584133 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.584134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.584338 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.584397 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.584398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.585024 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.585025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.585541 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.585542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.585921 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.585921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.585996 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.586413 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.586417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.586767 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.588365 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.588366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.588366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.588532 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.588742 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.588743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.588743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.588799 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.589047 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.589048 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.589091 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.589175 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.589274 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260929 03:27:06.589367 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.583898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:57735 (local address 127.6.116.190:45013)
0929 03:27:06.583940 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.583942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.583998 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:27:06.584067 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.584067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.584077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.584127 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.584475 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.584929 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.585636 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.585838 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.585919 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.586857 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.586858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.586858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.586865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.588265 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.588618 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.588619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.588620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.588670 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.588868 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.588869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.588870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.588977 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.589045 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.589220 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.589287 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.608680 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.602482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.6.116.190:45013)
0929 03:27:06.602537 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.602539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.602547 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:06.602590 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.602590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.602598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.602645 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.603203 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.603659 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.604314 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.604482 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.604528 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.606472 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.606473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.606474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.606479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.607866 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.608146 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.608147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.608147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.608198 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.608348 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.608349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.608349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.608430 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.608492 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.608571 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.608619 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:27:06.610137 7734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:06.611737 7734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:06.617755 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.6.116.129:33807)
0929 03:27:06.614098 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.614102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.614161 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:27:06.614245 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.614247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.614263 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.614344 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.614551 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615383 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.616749 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617319 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.617415 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.617472 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.617632 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.617669 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260929 03:27:06.617754 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:33807 (local address 127.0.0.1:40000)
0929 03:27:06.614099 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.614110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.614180 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.614407 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.614408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.614410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.614472 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.614474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615466 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.615467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.616657 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.616660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617429 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.617430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.617511 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.617531 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.617615 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.617669 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260929 03:27:06.618072 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.6.116.130:45183)
0929 03:27:06.614088 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.614093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.614168 (+ 75us) server_negotiation.cc:407] Connection header received
0929 03:27:06.614247 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.614249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.614263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.614344 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.614531 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615738 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.616930 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617591 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.617650 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.617713 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.617928 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.617968 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260929 03:27:06.618085 8236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45809 (local address 127.0.0.1:54430)
0929 03:27:06.613809 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.613846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.613909 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.614510 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.614511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.614513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.614545 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.614546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615876 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.615877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.616722 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.616724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617654 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.617655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.617713 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.617717 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.617749 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.617787 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":64,"threads_started":1}
I20260929 03:27:06.618010 8238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45183 (local address 127.0.0.1:37100)
0929 03:27:06.614105 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.614114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.614173 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.614369 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.614372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.614374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.614447 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.614449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615832 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.615833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.616857 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.616859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617743 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.617744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.617799 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.617802 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.617850 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.617890 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260929 03:27:06.618546 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.613866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.6.116.131:45809)
0929 03:27:06.614294 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.614299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.614335 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:27:06.614370 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.614371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.614386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.614662 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.614719 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.615779 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.616811 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.617551 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.618054 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.618390 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.618415 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.618454 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":88,"threads_started":1}
I20260929 03:27:06.618451 7874 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 (DEFAULT_TABLE table=test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.618794 7874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2f22dceaf41cdb42c4679723016a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.618863 8015 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 (DEFAULT_TABLE table=test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.619187 8015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2f22dceaf41cdb42c4679723016a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.619676 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 (DEFAULT_TABLE table=test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.619890 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ba2f22dceaf41cdb42c4679723016a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.622402 8242 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Bootstrap starting.
I20260929 03:27:06.622604 8244 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Bootstrap starting.
I20260929 03:27:06.622748 8243 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Bootstrap starting.
I20260929 03:27:06.623365 8244 tablet_bootstrap.cc:654] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.623481 8243 tablet_bootstrap.cc:654] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.623576 8242 tablet_bootstrap.cc:654] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.623701 8244 log.cc:824] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:06.623797 8243 log.cc:824] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:06.623876 8242 log.cc:824] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:06.624454 8244 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: No bootstrap required, opened a new log
I20260929 03:27:06.624524 8244 ts_tablet_manager.cc:1397] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:06.624552 8242 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: No bootstrap required, opened a new log
I20260929 03:27:06.624619 8242 ts_tablet_manager.cc:1397] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:06.626523 8244 raft_consensus.cc:348] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.626523 8242 raft_consensus.cc:348] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.626685 8244 raft_consensus.cc:374] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.626685 8242 raft_consensus.cc:374] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.626722 8242 raft_consensus.cc:729] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4bf86393b94541a4bfc18742e6b141, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.626722 8244 raft_consensus.cc:729] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995d486e47c3493e824d276e0347c863, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.626845 8242 consensus_queue.cc:260] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.626868 8244 consensus_queue.cc:260] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.627127 8242 ts_tablet_manager.cc:1428] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:27:06.627127 8244 ts_tablet_manager.cc:1428] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:06.627163 8222 heartbeater.cc:503] Master 127.6.116.190:45013 was elected leader, sending a full tablet report...
I20260929 03:27:06.627526 8243 tablet_bootstrap.cc:492] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: No bootstrap required, opened a new log
I20260929 03:27:06.627529 7940 heartbeater.cc:503] Master 127.6.116.190:45013 was elected leader, sending a full tablet report...
I20260929 03:27:06.627588 8243 ts_tablet_manager.cc:1397] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:27:06.629173 8243 raft_consensus.cc:348] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.629338 8243 raft_consensus.cc:374] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.629383 8243 raft_consensus.cc:729] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5dcaa00c2946d087d6e30553241261, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.629489 8243 consensus_queue.cc:260] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.629755 8243 ts_tablet_manager.cc:1428] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:06.629818 8081 heartbeater.cc:503] Master 127.6.116.190:45013 was elected leader, sending a full tablet report...
I20260929 03:27:06.632068 8249 raft_consensus.cc:482] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:06.632140 8249 raft_consensus.cc:504] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.632429 8249 leader_election.cc:288] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809), 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183)
I20260929 03:27:06.635208 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.632647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:35021 (local address 127.6.116.130:45183)
0929 03:27:06.632728 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.632729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.632740 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:27:06.632774 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.632774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.632784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.632849 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.632975 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.633426 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.634375 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.634872 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.634923 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.634929 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.635127 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.635169 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:27:06.635231 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.632585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45183 (local address 127.6.116.129:35021)
0929 03:27:06.632642 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.632652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.632689 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.632859 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.632860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.632861 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.632905 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.632906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.633505 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.633507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.634246 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.634249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.634941 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.634942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.635031 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.635034 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.635121 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.635157 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:27:06.635511 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.632642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43049 (local address 127.6.116.131:45809)
0929 03:27:06.632696 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.632698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.632872 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:27:06.632950 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.632951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.632958 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.633008 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.633158 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.633599 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.634716 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.635191 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.635255 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.635263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.635445 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.635473 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:27:06.635565 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.632585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45809 (local address 127.6.116.129:43049)
0929 03:27:06.632815 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.632826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.632895 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.633012 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.633012 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.633014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.633078 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.633080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.633698 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.633699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.634638 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.634641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.635278 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.635279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.635368 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.635372 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.635439 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.635473 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":92,"thread_start_us":75,"threads_started":1}
I20260929 03:27:06.635689 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2f22dceaf41cdb42c4679723016a7" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261" is_pre_election: true
I20260929 03:27:06.635808 8035 raft_consensus.cc:2454] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 0.
I20260929 03:27:06.635854 8176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2f22dceaf41cdb42c4679723016a7" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4bf86393b94541a4bfc18742e6b141" is_pre_election: true
I20260929 03:27:06.635978 8176 raft_consensus.cc:2454] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 0.
I20260929 03:27:06.636021 7824 leader_election.cc:302] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5dcaa00c2946d087d6e30553241261, 995d486e47c3493e824d276e0347c863; no voters:
I20260929 03:27:06.636149 8249 raft_consensus.cc:2781] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:06.636216 8249 raft_consensus.cc:482] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:06.636252 8249 raft_consensus.cc:3037] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.637105 8249 raft_consensus.cc:504] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.637243 8249 leader_election.cc:288] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 election: Requested vote from peers 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809), 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183)
I20260929 03:27:06.637367 8176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2f22dceaf41cdb42c4679723016a7" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4bf86393b94541a4bfc18742e6b141"
I20260929 03:27:06.637444 8176 raft_consensus.cc:3037] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.637426 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ba2f22dceaf41cdb42c4679723016a7" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261"
I20260929 03:27:06.637518 8035 raft_consensus.cc:3037] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.638083 8176 raft_consensus.cc:2454] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 1.
I20260929 03:27:06.638230 8035 raft_consensus.cc:2454] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 1.
I20260929 03:27:06.638250 7826 leader_election.cc:302] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4bf86393b94541a4bfc18742e6b141, 995d486e47c3493e824d276e0347c863; no voters:
I20260929 03:27:06.638362 8249 raft_consensus.cc:2781] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:06.638487 8249 raft_consensus.cc:686] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 1 LEADER]: Becoming Leader. State: Replica: 995d486e47c3493e824d276e0347c863, State: Running, Role: LEADER
I20260929 03:27:06.638567 8249 consensus_queue.cc:237] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } }
I20260929 03:27:06.639279 7735 catalog_manager.cc:5495] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 reported cstate change: term changed from 0 to 1, leader changed from <none> to 995d486e47c3493e824d276e0347c863 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:06.644963 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.642903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.6.116.129:33807)
0929 03:27:06.642956 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.642958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.642970 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:27:06.643083 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.643084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.643098 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.643177 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:06.643281 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.643687 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.644364 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.644525 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.644570 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.644736 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.644809 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.644859 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.644874 (+ 15us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.644889 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:06.646379 8176 raft_consensus.cc:1261] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Refusing update from remote peer 995d486e47c3493e824d276e0347c863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.646440 8035 raft_consensus.cc:1261] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Refusing update from remote peer 995d486e47c3493e824d276e0347c863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.646689 8252 consensus_queue.cc:1035] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.646802 8249 consensus_queue.cc:1035] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.648865 8255 mvcc.cc:205] Tried to move back new op lower bound from 7334512339541209088 to 7334512339512102912. Current Snapshot: MvccSnapshot[applied={T|T < 7334512339541209088}]
I20260929 03:27:06.648934 8258 mvcc.cc:205] Tried to move back new op lower bound from 7334512339541209088 to 7334512339512102912. Current Snapshot: MvccSnapshot[applied={T|T < 7334512339541209088}]
I20260929 03:27:06.652315 7735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:06.652398 7735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:06.653035 7735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Sending DeleteTablet for 3 replicas of tablet 0ba2f22dceaf41cdb42c4679723016a7
I20260929 03:27:06.653343 8015 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:37100
I20260929 03:27:06.653342 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:54430
I20260929 03:27:06.653540 7874 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ba2f22dceaf41cdb42c4679723016a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:40000
I20260929 03:27:06.653738 8268 tablet_replica.cc:330] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: stopping tablet replica
I20260929 03:27:06.653811 8266 tablet_replica.cc:330] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: stopping tablet replica
I20260929 03:27:06.653831 8268 raft_consensus.cc:2229] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:06.653936 8268 raft_consensus.cc:2258] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.654121 7735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:06.654275 7735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:06.654337 8268 ts_tablet_manager.cc:1905] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.654590 8266 raft_consensus.cc:2229] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:06.654708 8266 raft_consensus.cc:2258] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.655606 8266 ts_tablet_manager.cc:1905] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.655788 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba6137d50784e438610c15c2c15bc30 (DEFAULT_TABLE table=test-table [id=b86ddf2a35f34ef0b1205885d441b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.655889 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba6137d50784e438610c15c2c15bc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.655978 8268 ts_tablet_manager.cc:1918] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:06.656036 8268 log.cc:1196] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/0ba2f22dceaf41cdb42c4679723016a7
I20260929 03:27:06.656281 8268 ts_tablet_manager.cc:1939] T 0ba2f22dceaf41cdb42c4679723016a7 P 995d486e47c3493e824d276e0347c863: Deleting consensus metadata
I20260929 03:27:06.657138 8242 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Bootstrap starting.
I20260929 03:27:06.657279 7720 catalog_manager.cc:4841] TS 995d486e47c3493e824d276e0347c863 (127.6.116.129:33807): tablet 0ba2f22dceaf41cdb42c4679723016a7 (table test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]) successfully deleted
I20260929 03:27:06.657327 8266 ts_tablet_manager.cc:1918] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:06.657387 8266 log.cc:1196] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/0ba2f22dceaf41cdb42c4679723016a7
I20260929 03:27:06.657538 8242 tablet_bootstrap.cc:654] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.657593 8015 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba6137d50784e438610c15c2c15bc30 (DEFAULT_TABLE table=test-table [id=b86ddf2a35f34ef0b1205885d441b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.657625 8266 ts_tablet_manager.cc:1939] T 0ba2f22dceaf41cdb42c4679723016a7 P 6d4bf86393b94541a4bfc18742e6b141: Deleting consensus metadata
I20260929 03:27:06.657698 8015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba6137d50784e438610c15c2c15bc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.657986 7721 catalog_manager.cc:4841] TS 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809): tablet 0ba2f22dceaf41cdb42c4679723016a7 (table test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]) successfully deleted
I20260929 03:27:06.658084 8242 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: No bootstrap required, opened a new log
I20260929 03:27:06.658118 8242 ts_tablet_manager.cc:1397] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:06.658128 8267 tablet_replica.cc:330] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: stopping tablet replica
I20260929 03:27:06.658224 8267 raft_consensus.cc:2229] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:06.658242 8242 raft_consensus.cc:348] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.658313 8242 raft_consensus.cc:374] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.658332 8242 raft_consensus.cc:729] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4bf86393b94541a4bfc18742e6b141, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.658373 8242 consensus_queue.cc:260] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.658464 8242 ts_tablet_manager.cc:1428] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.658550 8267 raft_consensus.cc:2258] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.658885 8243 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Bootstrap starting.
I20260929 03:27:06.659166 7874 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba6137d50784e438610c15c2c15bc30 (DEFAULT_TABLE table=test-table [id=b86ddf2a35f34ef0b1205885d441b75c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:06.659262 7874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba6137d50784e438610c15c2c15bc30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.660138 8244 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Bootstrap starting.
I20260929 03:27:06.659286 8243 tablet_bootstrap.cc:654] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.658952 8267 ts_tablet_manager.cc:1905] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.660562 8244 tablet_bootstrap.cc:654] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.660887 8243 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: No bootstrap required, opened a new log
I20260929 03:27:06.660933 8243 ts_tablet_manager.cc:1397] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:06.661057 8243 raft_consensus.cc:348] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.661113 8243 raft_consensus.cc:374] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.661132 8243 raft_consensus.cc:729] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5dcaa00c2946d087d6e30553241261, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.661168 8243 consensus_queue.cc:260] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.661242 8243 ts_tablet_manager.cc:1428] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.662266 8267 ts_tablet_manager.cc:1918] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:06.662319 8267 log.cc:1196] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/0ba2f22dceaf41cdb42c4679723016a7
I20260929 03:27:06.662485 8267 ts_tablet_manager.cc:1939] T 0ba2f22dceaf41cdb42c4679723016a7 P 0b5dcaa00c2946d087d6e30553241261: Deleting consensus metadata
I20260929 03:27:06.662887 8244 tablet_bootstrap.cc:492] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: No bootstrap required, opened a new log
I20260929 03:27:06.662933 8244 ts_tablet_manager.cc:1397] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:27:06.662927 7719 catalog_manager.cc:4841] TS 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183): tablet 0ba2f22dceaf41cdb42c4679723016a7 (table test-table [id=33d917c82b984ccd8047d9b7cdc96bd2]) successfully deleted
I20260929 03:27:06.663090 8244 raft_consensus.cc:348] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.663158 8244 raft_consensus.cc:374] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.663187 8244 raft_consensus.cc:729] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995d486e47c3493e824d276e0347c863, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.663231 8244 consensus_queue.cc:260] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.663326 8244 ts_tablet_manager.cc:1428] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 03:27:06.686939 8082 tablet.cc:2369] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:06.692675 8248 raft_consensus.cc:482] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:06.692739 8248 raft_consensus.cc:504] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.693049 8248 leader_election.cc:288] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 995d486e47c3493e824d276e0347c863 (127.6.116.129:33807), 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183)
I20260929 03:27:06.696414 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.693312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51235 (local address 127.6.116.129:33807)
0929 03:27:06.693390 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.693392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.693400 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:06.693425 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.693426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.693435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.693491 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.693730 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.694456 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.695705 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.696052 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.696147 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.696165 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.696324 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.696359 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:27:06.696413 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.693202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:33807 (local address 127.6.116.131:51235)
0929 03:27:06.693281 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.693292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.693334 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.693564 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.693565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.693566 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.693637 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.693639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.694553 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.694554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.695587 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.695590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.696146 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.696147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.696239 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.696242 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.696303 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.696346 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 03:27:06.696648 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.693507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51707 (local address 127.6.116.130:45183)
0929 03:27:06.693542 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.693545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.693551 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:06.693570 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.693571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.693578 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.693632 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:06.693815 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.694265 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.695685 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.696235 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.696327 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.696334 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.696550 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.696585 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:06.696769 7894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba6137d50784e438610c15c2c15bc30" candidate_uuid: "6d4bf86393b94541a4bfc18742e6b141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995d486e47c3493e824d276e0347c863" is_pre_election: true
I20260929 03:27:06.696874 7894 raft_consensus.cc:2454] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4bf86393b94541a4bfc18742e6b141 in term 0.
I20260929 03:27:06.696874 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.693232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45183 (local address 127.6.116.131:51707)
0929 03:27:06.693364 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.693373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.693413 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.693656 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.693657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.693658 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:06.693731 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.693732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.694351 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.694352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.695588 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.695590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.696395 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.696397 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.696470 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.696473 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.696745 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.696788 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260929 03:27:06.697078 8108 leader_election.cc:302] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4bf86393b94541a4bfc18742e6b141, 995d486e47c3493e824d276e0347c863; no voters:
I20260929 03:27:06.697187 8248 raft_consensus.cc:2781] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:06.697217 8248 raft_consensus.cc:482] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:06.697242 8248 raft_consensus.cc:3037] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.697224 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba6137d50784e438610c15c2c15bc30" candidate_uuid: "6d4bf86393b94541a4bfc18742e6b141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261" is_pre_election: true
I20260929 03:27:06.697299 8035 raft_consensus.cc:2454] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4bf86393b94541a4bfc18742e6b141 in term 0.
I20260929 03:27:06.697916 8248 raft_consensus.cc:504] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.698050 8248 leader_election.cc:288] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [CANDIDATE]: Term 1 election: Requested vote from peers 995d486e47c3493e824d276e0347c863 (127.6.116.129:33807), 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183)
I20260929 03:27:06.698189 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba6137d50784e438610c15c2c15bc30" candidate_uuid: "6d4bf86393b94541a4bfc18742e6b141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261"
I20260929 03:27:06.698263 8035 raft_consensus.cc:3037] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.698364 7894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ba6137d50784e438610c15c2c15bc30" candidate_uuid: "6d4bf86393b94541a4bfc18742e6b141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995d486e47c3493e824d276e0347c863"
I20260929 03:27:06.698438 7894 raft_consensus.cc:3037] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.698967 8035 raft_consensus.cc:2454] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4bf86393b94541a4bfc18742e6b141 in term 1.
I20260929 03:27:06.699169 8107 leader_election.cc:302] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5dcaa00c2946d087d6e30553241261, 6d4bf86393b94541a4bfc18742e6b141; no voters:
I20260929 03:27:06.699275 8248 raft_consensus.cc:2781] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:06.699276 7894 raft_consensus.cc:2454] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4bf86393b94541a4bfc18742e6b141 in term 1.
I20260929 03:27:06.699399 8248 raft_consensus.cc:686] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4bf86393b94541a4bfc18742e6b141, State: Running, Role: LEADER
I20260929 03:27:06.699476 8248 consensus_queue.cc:237] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.700150 7735 catalog_manager.cc:5495] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "6d4bf86393b94541a4bfc18742e6b141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } health_report { overall_health: HEALTHY } } }
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:06.734831 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.727671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45013 (local address 127.0.0.1:48420)
0929 03:27:06.727831 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.727849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.727907 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.728037 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.728038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.728310 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:06.728367 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.728368 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.729302 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.729302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.729814 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.729815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.730272 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.730273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.730319 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.730843 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:06.730847 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:06.732171 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:06.733854 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.733860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.733861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.734039 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.734253 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:06.734254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:06.734254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:06.734305 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:06.734574 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:06.734580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:06.734617 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.734679 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.734761 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260929 03:27:06.735149 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.727760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.6.116.190:45013)
0929 03:27:06.727810 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.727812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.727916 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:27:06.727944 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.727945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.727955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.728027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.728458 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.729215 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.729901 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.730185 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.730259 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.732262 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.732264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.732264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.732271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.733766 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.734117 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.734118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.734118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.734181 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.734370 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.734371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.734372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.734505 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.734581 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.734722 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.734827 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":29}
I20260929 03:27:06.737041 7735 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:48420:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:27:06.737262 7735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:06.738669 7874 tablet_service.cc:1467] Processing CreateTablet for tablet 8dceec7da421498a82798cac91b9bd6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e30af765a89e4872a18aebe6ff1b86eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:06.738710 8015 tablet_service.cc:1467] Processing CreateTablet for tablet 8dceec7da421498a82798cac91b9bd6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e30af765a89e4872a18aebe6ff1b86eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:06.738763 8156 tablet_service.cc:1467] Processing CreateTablet for tablet 8dceec7da421498a82798cac91b9bd6d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e30af765a89e4872a18aebe6ff1b86eb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:06.738822 8015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dceec7da421498a82798cac91b9bd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.738821 7874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dceec7da421498a82798cac91b9bd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.738910 8156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dceec7da421498a82798cac91b9bd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:06.740397 8244 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: Bootstrap starting.
I20260929 03:27:06.740397 8243 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: Bootstrap starting.
I20260929 03:27:06.740473 8242 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: Bootstrap starting.
I20260929 03:27:06.740801 8244 tablet_bootstrap.cc:654] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.740801 8243 tablet_bootstrap.cc:654] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.740897 8242 tablet_bootstrap.cc:654] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:06.741503 8243 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: No bootstrap required, opened a new log
I20260929 03:27:06.741554 8243 ts_tablet_manager.cc:1397] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:06.741689 8242 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: No bootstrap required, opened a new log
I20260929 03:27:06.741730 8242 ts_tablet_manager.cc:1397] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:06.741706 8243 raft_consensus.cc:348] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.741782 8243 raft_consensus.cc:374] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.741806 8243 raft_consensus.cc:729] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5dcaa00c2946d087d6e30553241261, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.741829 8244 tablet_bootstrap.cc:492] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: No bootstrap required, opened a new log
I20260929 03:27:06.741868 8244 ts_tablet_manager.cc:1397] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:06.741846 8243 consensus_queue.cc:260] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.741871 8242 raft_consensus.cc:348] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.741930 8242 raft_consensus.cc:374] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.741948 8242 raft_consensus.cc:729] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4bf86393b94541a4bfc18742e6b141, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.741945 8243 ts_tablet_manager.cc:1428] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.741992 8242 consensus_queue.cc:260] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.741997 8244 raft_consensus.cc:348] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.742067 8242 ts_tablet_manager.cc:1428] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:06.742074 8244 raft_consensus.cc:374] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:06.742095 8244 raft_consensus.cc:729] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995d486e47c3493e824d276e0347c863, State: Initialized, Role: FOLLOWER
I20260929 03:27:06.742130 8244 consensus_queue.cc:260] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.742211 8244 ts_tablet_manager.cc:1428] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:06.742188 8250 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.742272 8250 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.742326 8261 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.742395 8261 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.742376 8248 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.742439 8248 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.757500 8261 raft_consensus.cc:482] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:06.757581 8261 raft_consensus.cc:504] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.757740 8261 leader_election.cc:288] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183), 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809)
I20260929 03:27:06.757931 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dceec7da421498a82798cac91b9bd6d" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261" is_pre_election: true
I20260929 03:27:06.757930 8176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dceec7da421498a82798cac91b9bd6d" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4bf86393b94541a4bfc18742e6b141" is_pre_election: true
I20260929 03:27:06.758023 8035 raft_consensus.cc:2454] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 0.
I20260929 03:27:06.758033 8176 raft_consensus.cc:2454] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 0.
I20260929 03:27:06.758168 7824 leader_election.cc:302] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5dcaa00c2946d087d6e30553241261, 995d486e47c3493e824d276e0347c863; no voters:
I20260929 03:27:06.758258 8261 raft_consensus.cc:2781] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:06.758289 8261 raft_consensus.cc:482] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:06.758312 8261 raft_consensus.cc:3037] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.759106 8261 raft_consensus.cc:504] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.759224 8261 leader_election.cc:288] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 election: Requested vote from peers 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183), 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809)
I20260929 03:27:06.759379 8035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dceec7da421498a82798cac91b9bd6d" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5dcaa00c2946d087d6e30553241261"
I20260929 03:27:06.759433 8176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dceec7da421498a82798cac91b9bd6d" candidate_uuid: "995d486e47c3493e824d276e0347c863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4bf86393b94541a4bfc18742e6b141"
I20260929 03:27:06.759473 8035 raft_consensus.cc:3037] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.759493 8176 raft_consensus.cc:3037] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:06.760118 8035 raft_consensus.cc:2454] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 1.
I20260929 03:27:06.760217 8176 raft_consensus.cc:2454] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 995d486e47c3493e824d276e0347c863 in term 1.
I20260929 03:27:06.760274 7824 leader_election.cc:302] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5dcaa00c2946d087d6e30553241261, 995d486e47c3493e824d276e0347c863; no voters:
I20260929 03:27:06.760387 8261 raft_consensus.cc:2781] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:06.760465 8261 raft_consensus.cc:686] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [term 1 LEADER]: Becoming Leader. State: Replica: 995d486e47c3493e824d276e0347c863, State: Running, Role: LEADER
I20260929 03:27:06.760514 8261 consensus_queue.cc:237] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } }
I20260929 03:27:06.760783 8249 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: TxnStatusTablet state changed. Reason: New leader 995d486e47c3493e824d276e0347c863. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.760885 8249 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:06.761068 8285 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:27:06.761169 8285 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:27:06.761279 7733 catalog_manager.cc:5495] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 reported cstate change: term changed from 0 to 1, leader changed from <none> to 995d486e47c3493e824d276e0347c863 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:06.768015 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.765413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.6.116.129:33807)
0929 03:27:06.765464 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.765465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.765559 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:27:06.765634 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.765635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.765642 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.765683 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:06.765798 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.766244 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.767417 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.767588 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.767638 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.767834 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.767859 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.767892 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.767955 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.767971 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:27:06.768074 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.765349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:33807 (local address 127.0.0.1:40014)
0929 03:27:06.765491 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.765506 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.765568 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.765698 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.765699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.765700 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:06.765746 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.765748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.766329 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.766331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.767340 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.767344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.767665 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.767666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.767737 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.767753 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.767896 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.767902 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.767953 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.767997 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 03:27:06.768929 8035 raft_consensus.cc:1261] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Refusing update from remote peer 995d486e47c3493e824d276e0347c863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.768929 8176 raft_consensus.cc:1261] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Refusing update from remote peer 995d486e47c3493e824d276e0347c863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.769111 8249 consensus_queue.cc:1035] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.769188 8261 consensus_queue.cc:1035] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.769678 8248 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: TxnStatusTablet state changed. Reason: New leader 995d486e47c3493e824d276e0347c863. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.769773 8248 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.769770 8250 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: TxnStatusTablet state changed. Reason: New leader 995d486e47c3493e824d276e0347c863. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.769851 8250 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.770426 8261 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.770501 8261 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:06.770520 8252 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.770591 8252 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 995d486e47c3493e824d276e0347c863: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:06.770959 8257 mvcc.cc:205] Tried to move back new op lower bound from 7334512340044070912 to 7334512340011270144. Current Snapshot: MvccSnapshot[applied={T|T < 7334512340044070912}]
I20260929 03:27:06.771081 8250 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.771139 8250 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 0b5dcaa00c2946d087d6e30553241261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.771214 8248 tablet_replica.cc:439] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "995d486e47c3493e824d276e0347c863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 } } peers { permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 } } peers { permanent_uuid: "6d4bf86393b94541a4bfc18742e6b141" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45809 } } }
I20260929 03:27:06.771284 8248 tablet_replica.cc:442] T 8dceec7da421498a82798cac91b9bd6d P 6d4bf86393b94541a4bfc18742e6b141: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:06.774454 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.772026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.6.116.131:45809)
0929 03:27:06.772092 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.772094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.772100 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:06.772139 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.772139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.772158 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.772234 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:06.772337 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.772927 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.773650 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.773912 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.773988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.774122 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.774232 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.774311 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.774361 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.774382 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 03:27:06.778786 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.776649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:52537 (local address 127.6.116.129:33807)
0929 03:27:06.776699 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.776701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.776710 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:06.776764 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.776765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.776775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.776827 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:06.776955 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.777391 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.778097 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.778297 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.778380 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.778553 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.778584 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.778649 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.778707 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.778724 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:06.778805 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.776588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:33807 (local address 127.6.116.131:52537)
0929 03:27:06.776654 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.776663 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.776703 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.776830 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.776831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.776832 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:06.776886 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.776887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.777472 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.777473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.778016 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.778017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.778381 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.778382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.778444 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.778464 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.778648 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.778650 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.778703 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.778741 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:27:06.780788 8035 raft_consensus.cc:1261] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Refusing update from remote peer 6d4bf86393b94541a4bfc18742e6b141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.780840 7894 raft_consensus.cc:1261] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Refusing update from remote peer 6d4bf86393b94541a4bfc18742e6b141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:06.781076 8273 consensus_queue.cc:1035] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5dcaa00c2946d087d6e30553241261" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.781248 8248 consensus_queue.cc:1035] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "995d486e47c3493e824d276e0347c863" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 33807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:06.788625 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.786328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45809 (local address 127.6.116.129:55295)
0929 03:27:06.786391 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:06.786401 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:06.786434 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:06.786574 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:06.786575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:06.786577 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:06.786611 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.786613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.787277 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.787278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.787823 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:06.787824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.788222 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.788223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:06.788276 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.788290 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.788480 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.788482 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:06.788522 (+ 40us) client_negotiation.cc:240] Negotiation successful
0929 03:27:06.788563 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:27:06.788621 8239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.786381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:55295 (local address 127.6.116.131:45809)
0929 03:27:06.786414 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.786415 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.786438 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:27:06.786505 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.786506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.786517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.786560 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:06.786683 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.787200 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.787925 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.788168 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.788223 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.788358 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:06.788390 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:06.788447 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.788540 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.788559 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:06.797640 7734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:06.797744 7734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:06.798328 7734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16d74335631d4776ab011df266cba19e: Sending DeleteTablet for 3 replicas of tablet 9ba6137d50784e438610c15c2c15bc30
I20260929 03:27:06.798584 7874 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba6137d50784e438610c15c2c15bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:40000
I20260929 03:27:06.798681 8268 tablet_replica.cc:330] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: stopping tablet replica
I20260929 03:27:06.798673 8015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba6137d50784e438610c15c2c15bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:37100
I20260929 03:27:06.798730 8268 raft_consensus.cc:2229] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:06.798763 8268 raft_consensus.cc:2258] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.798830 8267 tablet_replica.cc:330] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: stopping tablet replica
I20260929 03:27:06.798875 8267 raft_consensus.cc:2229] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:06.798902 8267 raft_consensus.cc:2258] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.799197 8268 ts_tablet_manager.cc:1905] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.799214 8267 ts_tablet_manager.cc:1905] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.800992 8268 ts_tablet_manager.cc:1918] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:06.801039 8268 log.cc:1196] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/9ba6137d50784e438610c15c2c15bc30
I20260929 03:27:06.801160 8156 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba6137d50784e438610c15c2c15bc30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:06 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:54430
I20260929 03:27:06.801255 8266 tablet_replica.cc:330] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: stopping tablet replica
I20260929 03:27:06.801275 8268 ts_tablet_manager.cc:1939] T 9ba6137d50784e438610c15c2c15bc30 P 995d486e47c3493e824d276e0347c863: Deleting consensus metadata
I20260929 03:27:06.801296 8266 raft_consensus.cc:2229] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:06.801386 8266 raft_consensus.cc:2258] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:06.801586 7720 catalog_manager.cc:4841] TS 995d486e47c3493e824d276e0347c863 (127.6.116.129:33807): tablet 9ba6137d50784e438610c15c2c15bc30 (table test-table [id=b86ddf2a35f34ef0b1205885d441b75c]) successfully deleted
I20260929 03:27:06.801759 8266 ts_tablet_manager.cc:1905] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:06.802935 8266 ts_tablet_manager.cc:1918] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:06.802971 8266 log.cc:1196] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/9ba6137d50784e438610c15c2c15bc30
I20260929 03:27:06.803193 8266 ts_tablet_manager.cc:1939] T 9ba6137d50784e438610c15c2c15bc30 P 6d4bf86393b94541a4bfc18742e6b141: Deleting consensus metadata
I20260929 03:27:06.803582 8267 ts_tablet_manager.cc:1918] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:06.803634 8267 log.cc:1196] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/9ba6137d50784e438610c15c2c15bc30
I20260929 03:27:06.803821 8267 ts_tablet_manager.cc:1939] T 9ba6137d50784e438610c15c2c15bc30 P 0b5dcaa00c2946d087d6e30553241261: Deleting consensus metadata
I20260929 03:27:06.804069 7719 catalog_manager.cc:4841] TS 0b5dcaa00c2946d087d6e30553241261 (127.6.116.130:45183): tablet 9ba6137d50784e438610c15c2c15bc30 (table test-table [id=b86ddf2a35f34ef0b1205885d441b75c]) successfully deleted
I20260929 03:27:06.804258 7721 catalog_manager.cc:4841] TS 6d4bf86393b94541a4bfc18742e6b141 (127.6.116.131:45809): tablet 9ba6137d50784e438610c15c2c15bc30 (table test-table [id=b86ddf2a35f34ef0b1205885d441b75c]) successfully deleted
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[7690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652426, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:06.813076 8241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.803356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40020 (local address 127.6.116.129:33807)
0929 03:27:06.803395 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.803397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.804314 (+ 917us) server_negotiation.cc:407] Connection header received
0929 03:27:06.804464 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.804465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.804475 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.804530 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.805107 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.805613 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.806673 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.806943 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.806983 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.810253 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.810262 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.810262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.810269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.811894 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.812275 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.812278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.812279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.812337 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.812541 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.812541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.812542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.812683 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.812828 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.812904 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.812981 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:27:06.813505 7934 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40020
I20260929 03:27:06.820349 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:06.814324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48428 (local address 127.6.116.190:45013)
0929 03:27:06.814364 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:06.814366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:06.814548 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:27:06.814617 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:06.814618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:06.814628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:06.814683 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:06.815224 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.815695 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.816373 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:06.816583 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:06.816656 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:06.817780 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:06.817781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:06.817781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:06.817789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:06.819243 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.819596 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.819597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.819598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.819649 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:06.819865 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:06.819866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:06.819866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:06.819976 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:06.820078 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:06.820186 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:27:06.820254 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:27:06.820649 7734 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48428
I20260929 03:27:06.821249 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7808
I20260929 03:27:06.828878 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7949
I20260929 03:27:06.834407 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 8090
I20260929 03:27:06.838580 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 7703
2026-09-29T03:27:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2003 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[8302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[8302](info): set up 2 sockets
Sep 29 03:27:06 dist-test-slave-z2zg krb5kdc[8302](info): commencing operation
krb5kdc: starting...
W20260929 03:27:07.868513 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:07 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:07.978068 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36921
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:08.055483 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:08.055673 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:08.055696 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:08.056835 8315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:08.056874 8315 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:27:08.056886 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:08.056900 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:08.056911 8315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:08.056964 8315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:08.058387 8315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:36921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:08.058624 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:08.058844 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:08.063544 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:08.063549 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.063723 8315 server_base.cc:1034] running on GCE node
W20260929 03:27:08.063655 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.063989 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:08.064256 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:08.065413 8315 hybrid_clock.cc:648] HybridClock initialized: now 1790652428065388 us; error 55 us; skew 500 ppm
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:08.067737 8315 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:08.068127 8315 webserver.cc:460] Webserver started at http://127.6.116.190:36065/ using document root <none> and password file <none>
I20260929 03:27:08.068346 8315 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:08.068411 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:08.068549 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:08.069512 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "8231abadf06cb0a476366c6b55b54d5a"
server_key_iv: "f54aafac7aeb920e480c0a763aefc786"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.069859 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "8231abadf06cb0a476366c6b55b54d5a"
server_key_iv: "f54aafac7aeb920e480c0a763aefc786"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.071107 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:08.071825 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.072014 8315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:08.072106 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "8231abadf06cb0a476366c6b55b54d5a"
server_key_iv: "f54aafac7aeb920e480c0a763aefc786"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.072186 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:08.085707 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:08.087443 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:08.087591 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:08.091868 8315 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:36921
I20260929 03:27:08.091917 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:36921 every 8 connection(s)
I20260929 03:27:08.092222 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:08.092804 8394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:08.094842 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Bootstrap starting.
I20260929 03:27:08.095470 8394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:08.095746 8394 log.cc:824] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:08.096392 8394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: No bootstrap required, opened a new log
I20260929 03:27:08.098316 8394 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } }
I20260929 03:27:08.098460 8394 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:08.098505 8394 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5869f82e6ba4b358a9deeb1c7fa4d66, State: Initialized, Role: FOLLOWER
I20260929 03:27:08.098682 8394 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } }
I20260929 03:27:08.098778 8394 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:08.098819 8394 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:08.098867 8394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:08.099421 8394 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } }
I20260929 03:27:08.099531 8394 leader_election.cc:302] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5869f82e6ba4b358a9deeb1c7fa4d66; no voters:
I20260929 03:27:08.099699 8394 leader_election.cc:288] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:08.099717 8397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:08.099850 8397 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [term 1 LEADER]: Becoming Leader. State: Replica: c5869f82e6ba4b358a9deeb1c7fa4d66, State: Running, Role: LEADER
I20260929 03:27:08.099927 8394 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:08.099973 8397 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } }
I20260929 03:27:08.100355 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5869f82e6ba4b358a9deeb1c7fa4d66. Latest consensus state: current_term: 1 leader_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } } }
I20260929 03:27:08.100438 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:08.100347 8398 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5869f82e6ba4b358a9deeb1c7fa4d66" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36921 } } }
I20260929 03:27:08.100654 8398 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:08.100720 8406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:08.101176 8406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:08.102190 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 8315
I20260929 03:27:08.102293 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:08.102448 6610 external_mini_cluster.cc:1428] Setting key a81b8187da469a8e5c1c46417f9f6770
I20260929 03:27:08.102610 8406 catalog_manager.cc:1347] Generated new cluster ID: 46f0f3ad3bba486baf81ce61f02a619b
I20260929 03:27:08.102679 8406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:08.115731 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.103059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.6.116.190:36921)
0929 03:27:08.107167 (+ 4108us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.107174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.107198 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:27:08.107283 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.107288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.107327 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.107460 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.108099 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.109014 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.109752 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.110138 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.110273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.112543 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.112547 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.112547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.112564 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.114165 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.114535 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.114538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.114538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.114622 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.114842 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.114843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.114844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.115113 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.115254 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.115370 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.115516 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":4056,"thread_start_us":76,"threads_started":1}
I20260929 03:27:08.135284 8406 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:08.135788 8406 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:08.142787 8406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Generated new TSK 0
I20260929 03:27:08.142976 8406 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:08.179371 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36921
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:08.261145 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:08.261317 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:08.261338 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:08.262336 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:08.262368 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:08.262387 8420 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:27:08.262398 8420 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:08.262414 8420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:08.263792 8420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:08.263954 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:08.264107 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:08.268770 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.268772 8420 server_base.cc:1034] running on GCE node
W20260929 03:27:08.268743 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:08.268733 8426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.269203 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:08.269448 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:08.270581 8420 hybrid_clock.cc:648] HybridClock initialized: now 1790652428270558 us; error 38 us; skew 500 ppm
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:08.272897 8420 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:08.273249 8420 webserver.cc:460] Webserver started at http://127.6.116.129:38495/ using document root <none> and password file <none>
I20260929 03:27:08.273414 8420 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:08.273455 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:08.273591 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:08.274680 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "47da0c76896643d8a5727f56d334830c"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "aa75f8261c6b0b59d56bb49229eabb63"
server_key_iv: "effbf1baa66a5bc62aaa1b1787606fdc"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.275002 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "47da0c76896643d8a5727f56d334830c"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "aa75f8261c6b0b59d56bb49229eabb63"
server_key_iv: "effbf1baa66a5bc62aaa1b1787606fdc"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.276289 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:08.277007 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.277199 8420 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:08.277293 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "47da0c76896643d8a5727f56d334830c"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "aa75f8261c6b0b59d56bb49229eabb63"
server_key_iv: "effbf1baa66a5bc62aaa1b1787606fdc"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.277372 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:08.288341 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:08.290400 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:08.290546 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:08.290977 8420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:08.291035 8420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.291095 8420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:08.291121 8420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.297926 8420 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:35803
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:08.298025 8553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:35803 every 8 connection(s)
I20260929 03:27:08.298293 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:08.301052 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.291457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:38319 (local address 127.6.116.190:36921)
0929 03:27:08.291525 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.291528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.291777 (+ 249us) server_negotiation.cc:407] Connection header received
0929 03:27:08.292438 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.292440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.292451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.292518 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.293331 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.294202 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.295626 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.295847 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.295932 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.298511 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.298513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.298513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.298523 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.299820 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.300224 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.300225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.300225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.300276 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.300528 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.300529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.300529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.300647 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.300699 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.300871 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.300932 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:27:08.301215 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.291440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.129:38319)
0929 03:27:08.291638 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.291687 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.292314 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.292584 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.292587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.292950 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.293237 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.293243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.294294 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.294295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.295522 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.295524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.295963 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.295964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.296083 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.296556 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.296563 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.298414 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.299887 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.299893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.299895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.300146 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.300363 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.300364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.300364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.300472 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.300689 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.300692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.300786 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.300857 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.300961 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":110,"threads_started":1}
I20260929 03:27:08.301681 8554 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36921
I20260929 03:27:08.301759 8554 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:08.301966 8554 heartbeater.cc:511] Master 127.6.116.190:36921 requested a full tablet report, sending...
I20260929 03:27:08.302533 8346 ts_manager.cc:194] Registered new tserver with Master: 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803)
I20260929 03:27:08.303258 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:38319
I20260929 03:27:08.303944 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 8420
I20260929 03:27:08.304049 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:08.304212 6610 external_mini_cluster.cc:1428] Setting key 805fd20c36412173ff419eb803c09149
I20260929 03:27:08.308511 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.302684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:58129 (local address 127.6.116.190:36921)
0929 03:27:08.302721 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.302723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.302855 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:27:08.302926 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.302927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.302936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.302990 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.303318 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.304017 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.304914 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.305163 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.305214 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.306126 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.306127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.306127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.306133 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.307492 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.307862 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.307863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.307863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.307908 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.308103 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.308104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.308104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.308194 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.308242 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.308403 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.308457 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:08.308642 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.302620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.129:58129)
0929 03:27:08.302790 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.302799 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.302869 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.302995 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.302995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.303188 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.303235 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.303236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.304127 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.304128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.304816 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.304818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.305232 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.305233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.305316 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.305660 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.305665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.306049 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.307570 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.307571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.307571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.307790 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.307970 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.307971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.307971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.308045 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.308255 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.308256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.308306 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.308380 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.308489 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:08.329802 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36921
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:08.410611 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:08.410825 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:08.410861 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:08.412004 8563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:08.412075 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:08.412117 8563 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:27:08.412142 8563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:08.412182 8563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:08.413548 8563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:08.413796 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:08.414022 8563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:08.418829 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.418859 8563 server_base.cc:1034] running on GCE node
W20260929 03:27:08.418817 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:08.418862 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.419263 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:08.419493 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:08.420634 8563 hybrid_clock.cc:648] HybridClock initialized: now 1790652428420623 us; error 26 us; skew 500 ppm
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:08.422890 8563 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:08.423364 8563 webserver.cc:460] Webserver started at http://127.6.116.130:46085/ using document root <none> and password file <none>
I20260929 03:27:08.423602 8563 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:08.423671 8563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:08.423800 8563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:08.424775 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "068bf27fc2c44250aae286e14417dc6e"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "0eeee9770f1309f5b044d10e97c0cfea"
server_key_iv: "f100faf3bdb5182ecfea8ec706b1b05d"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.425123 8563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "068bf27fc2c44250aae286e14417dc6e"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "0eeee9770f1309f5b044d10e97c0cfea"
server_key_iv: "f100faf3bdb5182ecfea8ec706b1b05d"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.426370 8563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:08.427146 8578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.427321 8563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:08.427413 8563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "068bf27fc2c44250aae286e14417dc6e"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "0eeee9770f1309f5b044d10e97c0cfea"
server_key_iv: "f100faf3bdb5182ecfea8ec706b1b05d"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.427505 8563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:08.437862 8563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:08.439607 8563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:08.439752 8563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:08.440306 8563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:08.440361 8563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.440397 8563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:08.440428 8563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:08.447741 8563 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45935
I20260929 03:27:08.448028 8694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45935 every 8 connection(s)
I20260929 03:27:08.448132 8563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:08.449802 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.440634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:49435 (local address 127.6.116.190:36921)
0929 03:27:08.440683 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.440686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.440952 (+ 266us) server_negotiation.cc:407] Connection header received
0929 03:27:08.441642 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.441643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.441655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.441699 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.442560 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.443295 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.444318 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.444520 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.444603 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.447064 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.447065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.447066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.447073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.448748 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.449083 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.449084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.449084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.449139 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.449359 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.449360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.449360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.449453 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.449506 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.449688 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.449740 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:08.450006 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.440627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.130:49435)
0929 03:27:08.440812 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.440856 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.441528 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.441769 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.441773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.442170 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.442463 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.442469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.443404 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.443405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.444194 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.444196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.444613 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.444613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.444778 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.445266 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.445276 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.446970 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.448811 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.448815 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.448817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.449012 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.449211 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.449211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.449212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.449306 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.449500 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.449502 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.449596 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.449673 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.449777 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260929 03:27:08.450474 8695 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36921
I20260929 03:27:08.450549 8695 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:08.450724 8695 heartbeater.cc:511] Master 127.6.116.190:36921 requested a full tablet report, sending...
I20260929 03:27:08.451077 8346 ts_manager.cc:194] Registered new tserver with Master: 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935)
I20260929 03:27:08.451469 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:49435
I20260929 03:27:08.455004 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 8563
I20260929 03:27:08.455124 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:08.455299 6610 external_mini_cluster.cc:1428] Setting key 24c4c35d253923df9a6efb24bdeae5c0
I20260929 03:27:08.457418 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.451468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.130:51785)
0929 03:27:08.451621 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.451631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.451707 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.452249 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.452250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.452427 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.452476 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.452478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.453091 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.453092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.453602 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.453604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.453986 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.453986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.454056 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.454477 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.454482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.454743 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.456392 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.456393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.456393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.456556 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.456760 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.456761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.456762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.456817 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.457164 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.457165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.457212 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.457284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.457359 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260929 03:27:08.457525 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.451552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:51785 (local address 127.6.116.190:36921)
0929 03:27:08.451592 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.451594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.451693 (+ 99us) server_negotiation.cc:407] Connection header received
0929 03:27:08.452161 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.452162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.452170 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.452245 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.452551 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.453001 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.453704 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.453903 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.453984 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.454848 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.454849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.454850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.454857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.456301 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.456628 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.456629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.456629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.456689 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.456888 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.456889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.456889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.457078 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.457158 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.457331 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.457410 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:08.477926 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36921
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:08.558223 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:08.558450 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:08.558490 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:08.559679 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:08.559749 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:08.559787 8704 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 03:27:08.559823 8704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:08.559867 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:08.561363 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:36921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:08.561637 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:08.561889 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:08.567692 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:08.567688 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:08.567746 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:08.568174 8704 server_base.cc:1034] running on GCE node
I20260929 03:27:08.568383 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:08.568657 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:08.569818 8704 hybrid_clock.cc:648] HybridClock initialized: now 1790652428569798 us; error 35 us; skew 500 ppm
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:08.572338 8704 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:08.572750 8704 webserver.cc:460] Webserver started at http://127.6.116.131:36001/ using document root <none> and password file <none>
I20260929 03:27:08.572973 8704 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:08.573014 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:08.573158 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:08.574258 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "65d9153553674671b3b50cb00a4f2e83"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "2a27450902b5810ab4dca97fd9990211"
server_key_iv: "f4c770605f8014508d4ed14e246dba76"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.574622 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "65d9153553674671b3b50cb00a4f2e83"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "2a27450902b5810ab4dca97fd9990211"
server_key_iv: "f4c770605f8014508d4ed14e246dba76"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.576030 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:08.576706 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.576875 8704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:08.576956 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "65d9153553674671b3b50cb00a4f2e83"
format_stamp: "Formatted at 2026-09-29 03:27:08 on dist-test-slave-z2zg"
server_key: "2a27450902b5810ab4dca97fd9990211"
server_key_iv: "f4c770605f8014508d4ed14e246dba76"
server_key_version: "encryptionkey@0"
I20260929 03:27:08.577005 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:08.592141 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:08.593860 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:08.594005 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:08.594446 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:08.594485 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.594507 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:08.594549 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:08.601305 8704 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:36131
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:08.601382 8840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:36131 every 8 connection(s)
I20260929 03:27:08.601773 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:08.602387 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 8704
I20260929 03:27:08.602490 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:08.602682 6610 external_mini_cluster.cc:1428] Setting key 000d6f23289fab209ef68355f3b3283b
I20260929 03:27:08.604692 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.594924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:34737 (local address 127.6.116.190:36921)
0929 03:27:08.594992 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.594994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.595161 (+ 167us) server_negotiation.cc:407] Connection header received
0929 03:27:08.595642 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.595643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.595655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.595702 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.596503 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.597297 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.598791 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.599161 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.599242 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.601905 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.601906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.601907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.601928 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.603630 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.603977 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.603977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.603978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.604028 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.604195 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.604195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.604195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.604290 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.604337 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.604524 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.604584 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:27:08.604785 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.594856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.131:34737)
0929 03:27:08.595020 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.595084 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.595541 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.595766 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.595771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.596102 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.596351 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.596356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.597385 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.597386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.598692 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.598694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.599269 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.599270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.599379 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.599932 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.599941 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.601820 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.603705 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.603709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.603711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.603914 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.604067 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.604067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.604067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.604144 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.604329 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.604331 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.604425 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.604495 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.604599 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260929 03:27:08.605175 8841 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36921
I20260929 03:27:08.605254 8841 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:08.605437 8841 heartbeater.cc:511] Master 127.6.116.190:36921 requested a full tablet report, sending...
I20260929 03:27:08.605739 8346 ts_manager.cc:194] Registered new tserver with Master: 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:08.606220 8346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:34737
I20260929 03:27:08.610874 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.606222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:45035 (local address 127.6.116.190:36921)
0929 03:27:08.606272 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.606273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.606374 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:27:08.606433 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.606434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.606443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.606496 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.606790 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.607354 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.607981 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.608147 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.608196 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.609001 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.609002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.609002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.609007 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.610088 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.610354 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.610354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.610355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.610398 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.610534 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.610534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.610534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.610607 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.610657 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.610772 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.610821 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:08.610883 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.606146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.6.116.131:45035)
0929 03:27:08.606316 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.606326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.606386 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.606507 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.606508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.606672 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:08.606719 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.606721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.607433 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.607434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.607918 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.607919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.608212 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.608212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.608263 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.608677 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:08.608681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:08.608928 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:08.610150 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.610151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.610151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.610297 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.610452 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:08.610452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:08.610453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:08.610499 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:08.610652 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:08.610653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:08.610691 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.610757 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.610827 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260929 03:27:08.614205 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:27:08 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:08.635778 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.629280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.6.116.190:36921)
0929 03:27:08.629339 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.629341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.629348 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:08.629399 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.629399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.629406 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.629452 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:08.630034 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.630509 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.631248 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.631413 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.631463 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.633612 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:08.633613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:08.633613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:08.633619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:08.634864 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.635233 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.635234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.635234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.635284 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:08.635435 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:08.635435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:08.635435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:08.635520 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:08.635587 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.635666 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.635721 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:27:08.637240 8346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:08.638757 8346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:08.645126 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.641033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.6.116.130:45935)
0929 03:27:08.641196 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.641199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.641394 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:27:08.641452 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.641454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.641469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.641560 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:08.641810 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.642922 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644187 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.644682 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644776 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.644833 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.644991 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.645028 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260929 03:27:08.645251 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.641111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:35803 (local address 127.0.0.1:56044)
0929 03:27:08.641274 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.641283 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.641323 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.641871 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.641872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.641874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:08.641908 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.641909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.643173 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.643174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.644122 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.644124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.645019 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.645020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.645070 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.645072 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.645101 (+ 29us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.645135 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":35,"threads_started":1}
I20260929 03:27:08.645305 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.640976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57384 (local address 127.6.116.131:36131)
0929 03:27:08.641140 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.641146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.641394 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:27:08.641448 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.641450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.641466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.641533 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:08.641718 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.642500 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644014 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.644743 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644812 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.644861 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.645175 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.645213 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260929 03:27:08.645164 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.640907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.0.0.1:53486)
0929 03:27:08.641192 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.641232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.641304 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.641670 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.641671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.641673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:08.641730 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.641732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.643012 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.643013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.644097 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.644099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.644787 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.644867 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.644879 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.644974 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.645028 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":51,"threads_started":1}
I20260929 03:27:08.645431 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.641312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.6.116.129:35803)
0929 03:27:08.641488 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:08.641492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:08.641503 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:27:08.641536 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:08.641538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:08.641552 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:08.641976 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:08.642029 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.643096 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644213 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.644929 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.645205 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.645263 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:08.645287 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:27:08.645326 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260929 03:27:08.645552 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:08.640915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:36131 (local address 127.0.0.1:57384)
0929 03:27:08.641241 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:08.641253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:08.641316 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:08.641550 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:08.641552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:08.641554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:08.641634 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.641636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.642619 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.642620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.643933 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:08.643936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:08.644895 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:08.644896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:08.644968 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:08.644973 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:08.645020 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:27:08.645074 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":60,"threads_started":1}
I20260929 03:27:08.646174 8485 tablet_service.cc:1467] Processing CreateTablet for tablet b268373633e146748d5ed6e23e6b0757 (DEFAULT_TABLE table=test-table [id=90a1d445216a459685ff754f1fe41865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:08.646364 8770 tablet_service.cc:1467] Processing CreateTablet for tablet b268373633e146748d5ed6e23e6b0757 (DEFAULT_TABLE table=test-table [id=90a1d445216a459685ff754f1fe41865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:08.646562 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b268373633e146748d5ed6e23e6b0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:08.646663 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b268373633e146748d5ed6e23e6b0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:08.647404 8628 tablet_service.cc:1467] Processing CreateTablet for tablet b268373633e146748d5ed6e23e6b0757 (DEFAULT_TABLE table=test-table [id=90a1d445216a459685ff754f1fe41865]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:08.647717 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b268373633e146748d5ed6e23e6b0757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:08.649838 8863 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Bootstrap starting.
I20260929 03:27:08.650135 8864 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Bootstrap starting.
I20260929 03:27:08.650799 8865 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Bootstrap starting.
I20260929 03:27:08.650977 8864 tablet_bootstrap.cc:654] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:08.651110 8863 tablet_bootstrap.cc:654] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:08.651373 8864 log.cc:824] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:08.651402 8863 log.cc:824] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:08.651499 8865 tablet_bootstrap.cc:654] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:08.651796 8865 log.cc:824] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:08.652299 8863 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: No bootstrap required, opened a new log
I20260929 03:27:08.652375 8863 ts_tablet_manager.cc:1397] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 03:27:08.652510 8865 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: No bootstrap required, opened a new log
I20260929 03:27:08.652587 8865 ts_tablet_manager.cc:1397] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:08.653785 8863 raft_consensus.cc:348] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.653910 8863 raft_consensus.cc:374] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:08.653932 8863 raft_consensus.cc:729] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Initialized, Role: FOLLOWER
I20260929 03:27:08.653861 8865 raft_consensus.cc:348] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.654026 8865 raft_consensus.cc:374] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:08.654085 8865 raft_consensus.cc:729] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Initialized, Role: FOLLOWER
I20260929 03:27:08.654053 8863 consensus_queue.cc:260] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.654197 8865 consensus_queue.cc:260] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.654316 8841 heartbeater.cc:503] Master 127.6.116.190:36921 was elected leader, sending a full tablet report...
I20260929 03:27:08.654314 8863 ts_tablet_manager.cc:1428] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:27:08.654462 8865 ts_tablet_manager.cc:1428] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:27:08.654511 8695 heartbeater.cc:503] Master 127.6.116.190:36921 was elected leader, sending a full tablet report...
I20260929 03:27:08.654860 8864 tablet_bootstrap.cc:492] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: No bootstrap required, opened a new log
I20260929 03:27:08.654928 8864 ts_tablet_manager.cc:1397] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:27:08.656517 8864 raft_consensus.cc:348] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.656651 8864 raft_consensus.cc:374] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:08.656672 8864 raft_consensus.cc:729] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Initialized, Role: FOLLOWER
I20260929 03:27:08.656744 8864 consensus_queue.cc:260] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:08.656924 8864 ts_tablet_manager.cc:1428] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:08.657009 8554 heartbeater.cc:503] Master 127.6.116.190:36921 was elected leader, sending a full tablet report...
W20260929 03:27:08.698658 8696 tablet.cc:2369] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:08.799022 8555 tablet.cc:2369] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:08.852355 8842 tablet.cc:2369] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:09.054724 8869 raft_consensus.cc:482] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:09.054888 8869 raft_consensus.cc:504] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:09.055310 8869 leader_election.cc:288] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803)
I20260929 03:27:09.058156 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.055624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:35467 (local address 127.6.116.130:45935)
0929 03:27:09.055678 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.055680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.055698 (+ 18us) server_negotiation.cc:407] Connection header received
0929 03:27:09.055782 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.055783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.055793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.055870 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:09.056060 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.056532 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.057457 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.057811 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.057890 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.057898 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.058061 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.058098 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:27:09.058156 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.055569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.6.116.131:35467)
0929 03:27:09.055636 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.055650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.055704 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.055878 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.055880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.055883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:09.055978 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.055980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.056612 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.056613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.057358 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.057360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.057889 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.057890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.057969 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.057977 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.058044 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.058078 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:27:09.058442 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b268373633e146748d5ed6e23e6b0757" candidate_uuid: "65d9153553674671b3b50cb00a4f2e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e" is_pre_election: true
I20260929 03:27:09.058532 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.055815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:36903 (local address 127.6.116.129:35803)
0929 03:27:09.055893 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.055895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.056032 (+ 137us) server_negotiation.cc:407] Connection header received
0929 03:27:09.056086 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.056087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.056098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.056172 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:09.056354 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.056826 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.057745 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.058105 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.058203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.058212 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.058428 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.058466 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:27:09.058552 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.055744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:35803 (local address 127.6.116.131:36903)
0929 03:27:09.055939 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.055948 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.056025 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.056175 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.056176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.056177 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:09.056275 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.056277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.056934 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.056935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.057645 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.057647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.058221 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.058223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.058316 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.058321 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.058414 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.058452 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":122,"threads_started":1}
I20260929 03:27:09.058619 8649 raft_consensus.cc:2454] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d9153553674671b3b50cb00a4f2e83 in term 0.
I20260929 03:27:09.058828 8722 leader_election.cc:302] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 65d9153553674671b3b50cb00a4f2e83; no voters:
I20260929 03:27:09.058895 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b268373633e146748d5ed6e23e6b0757" candidate_uuid: "65d9153553674671b3b50cb00a4f2e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c" is_pre_election: true
I20260929 03:27:09.058976 8869 raft_consensus.cc:2781] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:09.059020 8869 raft_consensus.cc:482] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:09.059079 8506 raft_consensus.cc:2454] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d9153553674671b3b50cb00a4f2e83 in term 0.
I20260929 03:27:09.059103 8869 raft_consensus.cc:3037] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.060065 8869 raft_consensus.cc:504] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:09.060209 8869 leader_election.cc:288] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [CANDIDATE]: Term 1 election: Requested vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803)
I20260929 03:27:09.060350 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b268373633e146748d5ed6e23e6b0757" candidate_uuid: "65d9153553674671b3b50cb00a4f2e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e"
I20260929 03:27:09.060385 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b268373633e146748d5ed6e23e6b0757" candidate_uuid: "65d9153553674671b3b50cb00a4f2e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c"
I20260929 03:27:09.060441 8649 raft_consensus.cc:3037] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.060494 8506 raft_consensus.cc:3037] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.061189 8506 raft_consensus.cc:2454] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d9153553674671b3b50cb00a4f2e83 in term 1.
I20260929 03:27:09.061211 8649 raft_consensus.cc:2454] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d9153553674671b3b50cb00a4f2e83 in term 1.
I20260929 03:27:09.061350 8722 leader_election.cc:302] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 65d9153553674671b3b50cb00a4f2e83; no voters:
I20260929 03:27:09.061492 8869 raft_consensus.cc:2781] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:09.061666 8869 raft_consensus.cc:686] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 1 LEADER]: Becoming Leader. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Running, Role: LEADER
I20260929 03:27:09.061764 8869 consensus_queue.cc:237] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } }
I20260929 03:27:09.062673 8347 catalog_manager.cc:5495] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "65d9153553674671b3b50cb00a4f2e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } health_report { overall_health: UNKNOWN } } }
Sep 29 03:27:09 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652428, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:09.114457 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.107669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.6.116.190:36921)
0929 03:27:09.107734 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.107737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.107841 (+ 104us) server_negotiation.cc:407] Connection header received
0929 03:27:09.107880 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.107881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.107889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.107944 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:09.108353 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.108826 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.109498 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.109685 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.109739 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.111931 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:09.111933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:09.111933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:09.111939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:09.113449 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:09.113830 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:09.113831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:09.113832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:09.113885 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:09.114063 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:09.114064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:09.114064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:09.114166 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:09.114216 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.114330 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.114399 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260929 03:27:09.114607 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.107618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36921 (local address 127.0.0.1:48260)
0929 03:27:09.107767 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.107778 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.107841 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.107963 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.107963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.108247 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:09.108289 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.108291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.108893 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.108893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.109418 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.109419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.109751 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.109752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.109790 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.110300 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:09.110304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:09.111850 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:09.113541 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:09.113547 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:09.113549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:09.113741 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:09.113953 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:09.113954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:09.113954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:09.114007 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:09.114220 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:09.114225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:09.114264 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.114315 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.114464 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":67,"thread_start_us":90,"threads_started":1}
I20260929 03:27:09.116412 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:48260:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:27:09.116575 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:09.117934 8485 tablet_service.cc:1467] Processing CreateTablet for tablet 8a768afb5b6348f684756959e4ce45fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47f57c0fb050466bbb40033e742df41c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:09.117965 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 8a768afb5b6348f684756959e4ce45fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47f57c0fb050466bbb40033e742df41c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:09.118108 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a768afb5b6348f684756959e4ce45fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.118126 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a768afb5b6348f684756959e4ce45fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.119289 8864 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: Bootstrap starting.
I20260929 03:27:09.119411 8770 tablet_service.cc:1467] Processing CreateTablet for tablet 8a768afb5b6348f684756959e4ce45fc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=47f57c0fb050466bbb40033e742df41c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:09.119530 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a768afb5b6348f684756959e4ce45fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.119622 8864 tablet_bootstrap.cc:654] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.120143 8864 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: No bootstrap required, opened a new log
I20260929 03:27:09.120185 8864 ts_tablet_manager.cc:1397] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:09.120347 8864 raft_consensus.cc:348] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.120399 8864 raft_consensus.cc:374] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.120412 8864 raft_consensus.cc:729] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.120469 8864 consensus_queue.cc:260] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.120570 8864 ts_tablet_manager.cc:1428] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:09.120599 8871 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.120669 8871 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.120669 8863 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: Bootstrap starting.
I20260929 03:27:09.120711 8865 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: Bootstrap starting.
I20260929 03:27:09.121052 8865 tablet_bootstrap.cc:654] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.121054 8863 tablet_bootstrap.cc:654] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.121692 8863 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: No bootstrap required, opened a new log
I20260929 03:27:09.121692 8865 tablet_bootstrap.cc:492] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: No bootstrap required, opened a new log
I20260929 03:27:09.121742 8863 ts_tablet_manager.cc:1397] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:09.121742 8865 ts_tablet_manager.cc:1397] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:09.121881 8863 raft_consensus.cc:348] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.121901 8865 raft_consensus.cc:348] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.121946 8863 raft_consensus.cc:374] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.121958 8865 raft_consensus.cc:374] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.121969 8863 raft_consensus.cc:729] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.121984 8865 raft_consensus.cc:729] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.122013 8863 consensus_queue.cc:260] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.122031 8865 consensus_queue.cc:260] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.122107 8863 ts_tablet_manager.cc:1428] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:09.122121 8865 ts_tablet_manager.cc:1428] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:09.122219 8869 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.122294 8869 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.122262 8870 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.122324 8870 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.237948 8870 raft_consensus.cc:482] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:09.238044 8870 raft_consensus.cc:504] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.238337 8870 leader_election.cc:288] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:09.242273 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.238596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:45189 (local address 127.6.116.129:35803)
0929 03:27:09.238640 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.238643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.238702 (+ 59us) server_negotiation.cc:407] Connection header received
0929 03:27:09.238800 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.238801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.238808 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.238849 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:09.239072 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.239875 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.241465 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.241876 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.241946 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.241955 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.242171 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.242211 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:09.242292 8885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.238516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:35803 (local address 127.6.116.130:45189)
0929 03:27:09.238630 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.238641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.238725 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.238880 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.238881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.238882 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:09.238965 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.238967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.239961 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.239962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.241377 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.241380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.242010 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.242011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.242084 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.242089 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.242158 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.242199 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":44,"thread_start_us":57,"threads_started":1}
I20260929 03:27:09.242597 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a768afb5b6348f684756959e4ce45fc" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c" is_pre_election: true
I20260929 03:27:09.242697 8506 raft_consensus.cc:2454] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 0.
I20260929 03:27:09.242734 8887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.238516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:36131 (local address 127.6.116.130:47935)
0929 03:27:09.238671 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.238679 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.238724 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.238980 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.238981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.238982 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:09.239036 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.239061 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.239964 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.239964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.241691 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.241693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.242480 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.242481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.242548 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.242552 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.242601 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.242644 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":76,"spinlock_wait_cycles":14080,"thread_start_us":54,"threads_started":1}
I20260929 03:27:09.242748 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.238596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:47935 (local address 127.6.116.131:36131)
0929 03:27:09.238709 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.238711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.238719 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:09.238895 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.238895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.238903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.238955 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:09.239130 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.239880 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.241773 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.242294 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.242381 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.242391 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.242621 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.242670 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":75,"threads_started":1}
I20260929 03:27:09.242887 8581 leader_election.cc:302] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 47da0c76896643d8a5727f56d334830c; no voters:
I20260929 03:27:09.243036 8870 raft_consensus.cc:2781] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:09.243099 8870 raft_consensus.cc:482] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:09.243122 8870 raft_consensus.cc:3037] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.243228 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a768afb5b6348f684756959e4ce45fc" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83" is_pre_election: true
I20260929 03:27:09.243345 8792 raft_consensus.cc:2454] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 0.
I20260929 03:27:09.243878 8870 raft_consensus.cc:504] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.244012 8870 leader_election.cc:288] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 election: Requested vote from peers 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:09.244135 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a768afb5b6348f684756959e4ce45fc" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c"
I20260929 03:27:09.244211 8506 raft_consensus.cc:3037] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.244191 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a768afb5b6348f684756959e4ce45fc" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83"
I20260929 03:27:09.244269 8792 raft_consensus.cc:3037] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.244778 8506 raft_consensus.cc:2454] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 1.
I20260929 03:27:09.244946 8581 leader_election.cc:302] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 47da0c76896643d8a5727f56d334830c; no voters:
I20260929 03:27:09.244959 8792 raft_consensus.cc:2454] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 1.
I20260929 03:27:09.245049 8870 raft_consensus.cc:2781] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:09.245178 8870 raft_consensus.cc:686] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [term 1 LEADER]: Becoming Leader. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Running, Role: LEADER
I20260929 03:27:09.245261 8870 consensus_queue.cc:237] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.245666 8889 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: TxnStatusTablet state changed. Reason: New leader 068bf27fc2c44250aae286e14417dc6e. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.245764 8889 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:09.245932 8891 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:27:09.246034 8891 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:27:09.246064 8347 catalog_manager.cc:5495] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 068bf27fc2c44250aae286e14417dc6e (127.6.116.130). New cstate: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:09.265930 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.263563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.6.116.130:45935)
0929 03:27:09.263621 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.263623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.263632 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:09.263737 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.263738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.263755 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.263819 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.263944 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.264410 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.265167 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.265365 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.265459 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.265625 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.265724 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.265811 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.265851 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.265869 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:09.265971 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.263489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.0.0.1:53488)
0929 03:27:09.263560 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.263592 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.263642 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.263824 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.263825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.263826 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:09.263880 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.263882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.264494 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.264495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.265077 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.265078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.265450 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.265451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.265510 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.265528 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.265790 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.265798 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.265845 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.265896 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:27:09.267863 8792 raft_consensus.cc:1261] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Refusing update from remote peer 068bf27fc2c44250aae286e14417dc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.267874 8506 raft_consensus.cc:1261] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Refusing update from remote peer 068bf27fc2c44250aae286e14417dc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.268117 8870 consensus_queue.cc:1035] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:09.268270 8870 consensus_queue.cc:1035] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:09.269295 8869 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: TxnStatusTablet state changed. Reason: New leader 068bf27fc2c44250aae286e14417dc6e. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.269312 8871 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: TxnStatusTablet state changed. Reason: New leader 068bf27fc2c44250aae286e14417dc6e. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.269384 8869 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.269393 8871 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.270255 8870 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.270330 8870 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:09.270383 8894 mvcc.cc:205] Tried to move back new op lower bound from 7334512350278103040 to 7334512350189019136. Current Snapshot: MvccSnapshot[applied={T|T < 7334512350278103040}]
I20260929 03:27:09.270977 8869 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.271090 8869 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 65d9153553674671b3b50cb00a4f2e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.271494 8889 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.271570 8889 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 068bf27fc2c44250aae286e14417dc6e: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:09.271785 8871 tablet_replica.cc:439] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } } }
I20260929 03:27:09.271867 8871 tablet_replica.cc:442] T 8a768afb5b6348f684756959e4ce45fc P 47da0c76896643d8a5727f56d334830c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:09.272410 8895 mvcc.cc:205] Tried to move back new op lower bound from 7334512350278103040 to 7334512350189019136. Current Snapshot: MvccSnapshot[applied={T|T < 7334512350278103040}]
I20260929 03:27:09.272495 8896 mvcc.cc:205] Tried to move back new op lower bound from 7334512350278103040 to 7334512350189019136. Current Snapshot: MvccSnapshot[applied={T|T < 7334512350278103040 or (T in {7334512350278103040})}]
I20260929 03:27:09.275893 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.273158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.6.116.131:36131)
0929 03:27:09.273200 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.273202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.273311 (+ 109us) server_negotiation.cc:407] Connection header received
0929 03:27:09.273383 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.273384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.273396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.273447 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.273569 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.273998 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.275137 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.275319 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.275366 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.275576 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.275650 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.275752 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.275808 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.275828 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:27:09.280848 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.278122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40107 (local address 127.6.116.130:45935)
0929 03:27:09.278159 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.278161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.278170 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:09.278230 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.278230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.278238 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.278305 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.278427 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.278845 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.279980 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.280315 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.280374 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.280586 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.280618 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.280674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.280744 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.280788 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:09.280871 8908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.277986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.6.116.131:40107)
0929 03:27:09.278123 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.278132 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.278174 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.278311 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.278311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.278313 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:09.278369 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.278370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.278911 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.278912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.279852 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.279854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.280386 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.280387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.280452 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.280464 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.280688 (+ 224us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.280690 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.280741 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.280781 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260929 03:27:09.283016 8649 raft_consensus.cc:1261] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Refusing update from remote peer 65d9153553674671b3b50cb00a4f2e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.283159 8506 raft_consensus.cc:1261] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Refusing update from remote peer 65d9153553674671b3b50cb00a4f2e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.283289 8874 consensus_queue.cc:1035] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:09.283403 8869 consensus_queue.cc:1035] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:09.291242 8888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.288641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:47987 (local address 127.6.116.131:36131)
0929 03:27:09.288686 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.288687 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.288787 (+ 100us) server_negotiation.cc:407] Connection header received
0929 03:27:09.288848 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.288849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.288856 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.288892 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.289004 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.289395 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.290621 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.290798 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.290848 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.291026 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.291086 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.291130 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.291186 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.291200 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:27:09.291353 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.288612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:36131 (local address 127.6.116.130:47987)
0929 03:27:09.288736 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.288746 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.288798 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.288906 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.288907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.288908 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:09.288955 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.288956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.289462 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.289463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.290550 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.290553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.290875 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.290876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.290939 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.290952 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.291132 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.291133 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.291184 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.291221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
I20260929 03:27:09.300468 8347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:09.300571 8347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:09.301244 8347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Sending DeleteTablet for 3 replicas of tablet b268373633e146748d5ed6e23e6b0757
I20260929 03:27:09.301499 8770 tablet_service.cc:1514] Processing DeleteTablet for tablet b268373633e146748d5ed6e23e6b0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:09 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:57384
I20260929 03:27:09.301600 8628 tablet_service.cc:1514] Processing DeleteTablet for tablet b268373633e146748d5ed6e23e6b0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:09 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53486
I20260929 03:27:09.301843 8918 tablet_replica.cc:330] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: stopping tablet replica
I20260929 03:27:09.301859 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet b268373633e146748d5ed6e23e6b0757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:09 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:56044
I20260929 03:27:09.302006 8918 raft_consensus.cc:2229] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:09.302112 8918 raft_consensus.cc:2258] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:09.302237 8347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:09.302366 8347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:09.302743 8918 ts_tablet_manager.cc:1905] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:09.303406 8770 tablet_service.cc:1467] Processing CreateTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (DEFAULT_TABLE table=test-table [id=8aefe8c2184442dfb29874b2d7861878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:09.303521 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.304625 8918 ts_tablet_manager.cc:1918] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:09.304770 8918 log.cc:1196] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/b268373633e146748d5ed6e23e6b0757
I20260929 03:27:09.304762 8863 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Bootstrap starting.
I20260929 03:27:09.305055 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (DEFAULT_TABLE table=test-table [id=8aefe8c2184442dfb29874b2d7861878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:09.305159 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.305209 8863 tablet_bootstrap.cc:654] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.305183 8918 ts_tablet_manager.cc:1939] T b268373633e146748d5ed6e23e6b0757 P 068bf27fc2c44250aae286e14417dc6e: Deleting consensus metadata
I20260929 03:27:09.305660 8333 catalog_manager.cc:4841] TS 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935): tablet b268373633e146748d5ed6e23e6b0757 (table test-table [id=90a1d445216a459685ff754f1fe41865]) successfully deleted
I20260929 03:27:09.306352 8865 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Bootstrap starting.
I20260929 03:27:09.306721 8865 tablet_bootstrap.cc:654] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.307150 8485 tablet_service.cc:1467] Processing CreateTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (DEFAULT_TABLE table=test-table [id=8aefe8c2184442dfb29874b2d7861878]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:09.307261 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:09.308447 8864 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Bootstrap starting.
I20260929 03:27:09.308734 8864 tablet_bootstrap.cc:654] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:09.309478 8864 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: No bootstrap required, opened a new log
I20260929 03:27:09.309530 8864 ts_tablet_manager.cc:1397] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:09.309659 8864 raft_consensus.cc:348] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.309731 8864 raft_consensus.cc:374] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.309751 8864 raft_consensus.cc:729] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.309787 8864 consensus_queue.cc:260] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.309873 8864 ts_tablet_manager.cc:1428] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:09.310269 8865 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: No bootstrap required, opened a new log
I20260929 03:27:09.310315 8865 ts_tablet_manager.cc:1397] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 03:27:09.310443 8865 raft_consensus.cc:348] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.310510 8865 raft_consensus.cc:374] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.310530 8865 raft_consensus.cc:729] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.310570 8865 consensus_queue.cc:260] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.310654 8865 ts_tablet_manager.cc:1428] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:09.310849 8919 tablet_replica.cc:330] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: stopping tablet replica
I20260929 03:27:09.310940 8919 raft_consensus.cc:2229] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:09.310994 8919 raft_consensus.cc:2258] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:09.311357 8919 ts_tablet_manager.cc:1905] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:09.312670 8919 ts_tablet_manager.cc:1918] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:09.312736 8919 log.cc:1196] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/b268373633e146748d5ed6e23e6b0757
I20260929 03:27:09.312959 8919 ts_tablet_manager.cc:1939] T b268373633e146748d5ed6e23e6b0757 P 47da0c76896643d8a5727f56d334830c: Deleting consensus metadata
I20260929 03:27:09.313321 8333 catalog_manager.cc:4841] TS 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803): tablet b268373633e146748d5ed6e23e6b0757 (table test-table [id=90a1d445216a459685ff754f1fe41865]) successfully deleted
I20260929 03:27:09.313374 8917 tablet_replica.cc:330] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: stopping tablet replica
I20260929 03:27:09.313464 8917 raft_consensus.cc:2229] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:09.313555 8917 raft_consensus.cc:2258] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:09.313887 8917 ts_tablet_manager.cc:1905] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:09.315374 8917 ts_tablet_manager.cc:1918] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:09.315436 8917 log.cc:1196] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/b268373633e146748d5ed6e23e6b0757
I20260929 03:27:09.315443 8863 tablet_bootstrap.cc:492] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: No bootstrap required, opened a new log
I20260929 03:27:09.315529 8863 ts_tablet_manager.cc:1397] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.001s
I20260929 03:27:09.315688 8917 ts_tablet_manager.cc:1939] T b268373633e146748d5ed6e23e6b0757 P 65d9153553674671b3b50cb00a4f2e83: Deleting consensus metadata
I20260929 03:27:09.315662 8863 raft_consensus.cc:348] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.315719 8863 raft_consensus.cc:374] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:09.315740 8863 raft_consensus.cc:729] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Initialized, Role: FOLLOWER
I20260929 03:27:09.315778 8863 consensus_queue.cc:260] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.315869 8863 ts_tablet_manager.cc:1428] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:09.316022 8334 catalog_manager.cc:4841] TS 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131): tablet b268373633e146748d5ed6e23e6b0757 (table test-table [id=90a1d445216a459685ff754f1fe41865]) successfully deleted
I20260929 03:27:09.477195 8889 raft_consensus.cc:482] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:09.477306 8889 raft_consensus.cc:504] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.477502 8889 leader_election.cc:288] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:09.477689 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e4c9d6d67b4db6a94f4a51d69b67c1" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83" is_pre_election: true
I20260929 03:27:09.477756 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e4c9d6d67b4db6a94f4a51d69b67c1" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c" is_pre_election: true
I20260929 03:27:09.477815 8792 raft_consensus.cc:2454] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 0.
I20260929 03:27:09.477847 8506 raft_consensus.cc:2454] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 0.
I20260929 03:27:09.478040 8582 leader_election.cc:302] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 65d9153553674671b3b50cb00a4f2e83; no voters:
I20260929 03:27:09.478147 8889 raft_consensus.cc:2781] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:09.478184 8889 raft_consensus.cc:482] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:09.478236 8889 raft_consensus.cc:3037] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.478978 8889 raft_consensus.cc:504] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.479138 8889 leader_election.cc:288] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 election: Requested vote from peers 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:09.479303 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e4c9d6d67b4db6a94f4a51d69b67c1" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83"
I20260929 03:27:09.479324 8506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e4c9d6d67b4db6a94f4a51d69b67c1" candidate_uuid: "068bf27fc2c44250aae286e14417dc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47da0c76896643d8a5727f56d334830c"
I20260929 03:27:09.479395 8792 raft_consensus.cc:3037] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.479393 8506 raft_consensus.cc:3037] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:09.480073 8506 raft_consensus.cc:2454] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 1.
I20260929 03:27:09.480178 8792 raft_consensus.cc:2454] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 068bf27fc2c44250aae286e14417dc6e in term 1.
I20260929 03:27:09.480268 8581 leader_election.cc:302] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 47da0c76896643d8a5727f56d334830c; no voters:
I20260929 03:27:09.480351 8889 raft_consensus.cc:2781] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:09.480425 8889 raft_consensus.cc:686] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 1 LEADER]: Becoming Leader. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Running, Role: LEADER
I20260929 03:27:09.480484 8889 consensus_queue.cc:237] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:09.481091 8347 catalog_manager.cc:5495] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 068bf27fc2c44250aae286e14417dc6e (127.6.116.130). New cstate: current_term: 1 leader_uuid: "068bf27fc2c44250aae286e14417dc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:09.496706 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.494484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.6.116.130:45935)
0929 03:27:09.494518 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.494520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.494529 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:09.494589 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.494590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.494603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.494676 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.494775 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.495310 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.496024 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.496226 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.496297 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.496446 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.496478 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.496547 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.496604 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.496625 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:27:09.500717 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.498459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:59215 (local address 127.6.116.130:45935)
0929 03:27:09.498515 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:09.498517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:09.498526 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:09.498577 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:09.498578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:09.498599 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:09.498649 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:09.498799 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.499306 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.500050 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.500309 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.500379 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.500508 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.500535 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.500581 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:09.500635 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:27:09.500647 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:09.500761 8912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:09.498416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.6.116.130:59215)
0929 03:27:09.498463 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:09.498471 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:09.498531 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:09.498681 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:09.498682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:09.498683 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:09.498716 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.498721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.499388 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.499388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.499959 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:09.499961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:09.500377 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:09.500377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:09.500424 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:09.500436 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:09.500579 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:09.500581 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:09.500619 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 03:27:09.500666 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 03:27:09.502354 8506 raft_consensus.cc:1261] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Refusing update from remote peer 068bf27fc2c44250aae286e14417dc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.502414 8792 raft_consensus.cc:1261] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Refusing update from remote peer 068bf27fc2c44250aae286e14417dc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:09.502564 8870 consensus_queue.cc:1035] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:09.502648 8889 consensus_queue.cc:1035] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:27:09.506944 8894 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:27:11.024258 8346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:11.024362 8346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:11.025141 8346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Sending DeleteTablet for 3 replicas of tablet 18e4c9d6d67b4db6a94f4a51d69b67c1
I20260929 03:27:11.025391 8770 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:57384
I20260929 03:27:11.025403 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:56044
I20260929 03:27:11.025592 8628 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53486
I20260929 03:27:11.025695 8947 tablet_replica.cc:330] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: stopping tablet replica
I20260929 03:27:11.025748 8947 raft_consensus.cc:2229] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:11.025758 8949 tablet_replica.cc:330] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: stopping tablet replica
I20260929 03:27:11.025791 8947 raft_consensus.cc:2258] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.025804 8949 raft_consensus.cc:2229] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:11.025978 8949 raft_consensus.cc:2258] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.026024 8947 ts_tablet_manager.cc:1905] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.026119 8948 tablet_replica.cc:330] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: stopping tablet replica
I20260929 03:27:11.026212 8948 raft_consensus.cc:2229] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:11.026221 8949 ts_tablet_manager.cc:1905] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.026343 8948 raft_consensus.cc:2258] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.026638 8948 ts_tablet_manager.cc:1905] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.027940 8949 ts_tablet_manager.cc:1918] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:27:11.028002 8949 log.cc:1196] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/18e4c9d6d67b4db6a94f4a51d69b67c1
I20260929 03:27:11.028007 8947 ts_tablet_manager.cc:1918] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:27:11.028075 8947 log.cc:1196] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/18e4c9d6d67b4db6a94f4a51d69b67c1
I20260929 03:27:11.028293 8947 ts_tablet_manager.cc:1939] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 65d9153553674671b3b50cb00a4f2e83: Deleting consensus metadata
I20260929 03:27:11.028288 8949 ts_tablet_manager.cc:1939] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 068bf27fc2c44250aae286e14417dc6e: Deleting consensus metadata
I20260929 03:27:11.028546 8948 ts_tablet_manager.cc:1918] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:27:11.028627 8948 log.cc:1196] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/18e4c9d6d67b4db6a94f4a51d69b67c1
I20260929 03:27:11.028649 8334 catalog_manager.cc:4841] TS 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131): tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (table test-table [id=8aefe8c2184442dfb29874b2d7861878]) successfully deleted
I20260929 03:27:11.028834 8948 ts_tablet_manager.cc:1939] T 18e4c9d6d67b4db6a94f4a51d69b67c1 P 47da0c76896643d8a5727f56d334830c: Deleting consensus metadata
I20260929 03:27:11.029137 8333 catalog_manager.cc:4841] TS 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803): tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (table test-table [id=8aefe8c2184442dfb29874b2d7861878]) successfully deleted
I20260929 03:27:11.029268 8333 catalog_manager.cc:4841] TS 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935): tablet 18e4c9d6d67b4db6a94f4a51d69b67c1 (table test-table [id=8aefe8c2184442dfb29874b2d7861878]) successfully deleted
Sep 29 03:27:11 dist-test-slave-z2zg krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:27:11 dist-test-slave-z2zg krb5kdc[8302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652431, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:11.049050 8957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.042283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.6.116.190:36921)
0929 03:27:11.042433 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.042435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.042450 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:27:11.042519 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.042520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.042529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.042622 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:11.043175 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.043658 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.044366 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.044547 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.044594 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.046679 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:11.046680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:11.046681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:11.046689 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:11.048226 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:11.048503 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:11.048503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:11.048504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:11.048553 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:11.048697 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:11.048697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:11.048697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:11.048785 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:11.048850 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.048930 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.048986 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260929 03:27:11.050448 8346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:11.050578 8346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:11.051828 8485 tablet_service.cc:1467] Processing CreateTablet for tablet 27fb5a58dac649ee9533e61e712ec225 (DEFAULT_TABLE table=test-table [id=720e9bf581c5449e87c3768be891027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.051832 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 27fb5a58dac649ee9533e61e712ec225 (DEFAULT_TABLE table=test-table [id=720e9bf581c5449e87c3768be891027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.051913 8770 tablet_service.cc:1467] Processing CreateTablet for tablet 27fb5a58dac649ee9533e61e712ec225 (DEFAULT_TABLE table=test-table [id=720e9bf581c5449e87c3768be891027f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.051966 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fb5a58dac649ee9533e61e712ec225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.051967 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fb5a58dac649ee9533e61e712ec225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.052042 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27fb5a58dac649ee9533e61e712ec225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.052914 8958 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Bootstrap starting.
I20260929 03:27:11.053301 8959 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Bootstrap starting.
I20260929 03:27:11.053308 8958 tablet_bootstrap.cc:654] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.053473 8960 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Bootstrap starting.
I20260929 03:27:11.053647 8959 tablet_bootstrap.cc:654] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.053857 8960 tablet_bootstrap.cc:654] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.054118 8958 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: No bootstrap required, opened a new log
I20260929 03:27:11.054185 8958 ts_tablet_manager.cc:1397] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:11.054325 8958 raft_consensus.cc:348] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.054392 8958 raft_consensus.cc:374] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.054406 8958 raft_consensus.cc:729] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.054417 8960 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: No bootstrap required, opened a new log
I20260929 03:27:11.054471 8960 ts_tablet_manager.cc:1397] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:11.054466 8958 consensus_queue.cc:260] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.054613 8958 ts_tablet_manager.cc:1428] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:11.054572 8960 raft_consensus.cc:348] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.054659 8960 raft_consensus.cc:374] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.054680 8960 raft_consensus.cc:729] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.054747 8960 consensus_queue.cc:260] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.055154 8960 ts_tablet_manager.cc:1428] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:11.055171 8959 tablet_bootstrap.cc:492] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: No bootstrap required, opened a new log
I20260929 03:27:11.055230 8959 ts_tablet_manager.cc:1397] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:11.055320 8959 raft_consensus.cc:348] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.055403 8959 raft_consensus.cc:374] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.055428 8959 raft_consensus.cc:729] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.055471 8959 consensus_queue.cc:260] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.055647 8959 ts_tablet_manager.cc:1428] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:11.063884 8964 raft_consensus.cc:482] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:11.063951 8964 raft_consensus.cc:504] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.064306 8964 leader_election.cc:288] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:11.067113 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.064456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:46751 (local address 127.6.116.130:45935)
0929 03:27:11.064563 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.064565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.064619 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:27:11.064700 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.064701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.064712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.064790 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:11.064956 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.065404 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.066385 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.066738 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.066820 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.066828 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.066985 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.067025 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":61,"threads_started":1}
I20260929 03:27:11.067113 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.064409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.6.116.129:46751)
0929 03:27:11.064568 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:11.064579 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:11.064635 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:11.064806 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:11.064806 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:11.064809 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:11.064885 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.064887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.065485 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.065486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.066280 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.066283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.066818 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.066819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.066893 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.066897 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:11.066977 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:27:11.067013 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260929 03:27:11.067299 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fb5a58dac649ee9533e61e712ec225" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e" is_pre_election: true
I20260929 03:27:11.067369 8649 raft_consensus.cc:2454] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 0.
I20260929 03:27:11.067382 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.064479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:36131 (local address 127.6.116.129:44249)
0929 03:27:11.064667 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:11.064675 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:11.064749 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:11.064907 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:11.064908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:11.064910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:11.065235 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.065237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.065835 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.065836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.066532 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.066533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.067111 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.067112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.067185 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.067189 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:11.067245 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:27:11.067288 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260929 03:27:11.067535 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.064515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:44249 (local address 127.6.116.131:36131)
0929 03:27:11.064619 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.064621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.064735 (+ 114us) server_negotiation.cc:407] Connection header received
0929 03:27:11.064800 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.064801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.064811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.064905 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:11.065308 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.065748 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.066615 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.066968 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.067087 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.067095 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.067422 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.067472 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260929 03:27:11.067520 8438 leader_election.cc:302] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 47da0c76896643d8a5727f56d334830c; no voters:
I20260929 03:27:11.067638 8964 raft_consensus.cc:2781] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:11.067673 8964 raft_consensus.cc:482] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:11.067698 8964 raft_consensus.cc:3037] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.067731 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fb5a58dac649ee9533e61e712ec225" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83" is_pre_election: true
I20260929 03:27:11.067801 8792 raft_consensus.cc:2454] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 0.
I20260929 03:27:11.068310 8964 raft_consensus.cc:504] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.068435 8964 leader_election.cc:288] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 election: Requested vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:11.068544 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fb5a58dac649ee9533e61e712ec225" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e"
I20260929 03:27:11.068550 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27fb5a58dac649ee9533e61e712ec225" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83"
I20260929 03:27:11.068629 8649 raft_consensus.cc:3037] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.068626 8792 raft_consensus.cc:3037] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.069108 8649 raft_consensus.cc:2454] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 1.
I20260929 03:27:11.069113 8792 raft_consensus.cc:2454] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 1.
I20260929 03:27:11.069265 8438 leader_election.cc:302] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 068bf27fc2c44250aae286e14417dc6e, 47da0c76896643d8a5727f56d334830c; no voters:
I20260929 03:27:11.069370 8964 raft_consensus.cc:2781] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:11.069494 8964 raft_consensus.cc:686] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 1 LEADER]: Becoming Leader. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Running, Role: LEADER
I20260929 03:27:11.069567 8964 consensus_queue.cc:237] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.070178 8345 catalog_manager.cc:5495] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c reported cstate change: term changed from 0 to 1, leader changed from <none> to 47da0c76896643d8a5727f56d334830c (127.6.116.129). New cstate: current_term: 1 leader_uuid: "47da0c76896643d8a5727f56d334830c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:11.078485 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.076451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.6.116.129:35803)
0929 03:27:11.076579 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.076581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.076588 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:11.076627 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.076628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.076642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.076715 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:11.076814 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.077245 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.077901 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.078085 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.078135 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.078243 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:11.078339 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:11.078389 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.078423 (+ 34us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.078436 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
I20260929 03:27:11.082564 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.080644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:60807 (local address 127.6.116.130:45935)
0929 03:27:11.080695 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.080697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.080709 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:27:11.080781 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.080782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.080789 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.080841 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:11.080986 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.081400 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.082049 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.082226 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.082277 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.082408 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:11.082429 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:11.082465 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.082511 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.082525 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:27:11.082607 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.080518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45935 (local address 127.6.116.129:60807)
0929 03:27:11.080662 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:11.080673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:11.080722 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:11.080837 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:11.080837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:11.080838 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:11.080909 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.080911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.081456 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.081456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.081976 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.081977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.082290 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.082290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.082334 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.082345 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:11.082477 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:11.082479 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:11.082511 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:27:11.082538 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260929 03:27:11.084357 8649 raft_consensus.cc:1261] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Refusing update from remote peer 47da0c76896643d8a5727f56d334830c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:11.084376 8792 raft_consensus.cc:1261] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Refusing update from remote peer 47da0c76896643d8a5727f56d334830c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:11.084601 8964 consensus_queue.cc:1035] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [LEADER]: Connected to new peer: Peer: permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:11.084728 8970 consensus_queue.cc:1035] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:11.091207 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.088957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:54881 (local address 127.6.116.129:35803)
0929 03:27:11.088991 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:11.088993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:11.089087 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:27:11.089181 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:11.089181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:11.089189 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:11.089229 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:11.089323 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.089736 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.090430 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.090679 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.090738 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.090937 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:11.090959 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:11.091096 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:11.091112 (+ 16us) server_negotiation.cc:299] Negotiation successful
0929 03:27:11.091132 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:11.091231 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:11.088893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:35803 (local address 127.6.116.130:54881)
0929 03:27:11.089036 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:11.089045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:11.089098 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:11.089242 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:11.089243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:11.089245 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:11.089279 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.089280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.089814 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.089815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.090337 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:11.090338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:11.090751 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:11.090752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:11.090812 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:11.090817 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:11.091066 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:11.091067 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:11.091106 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 03:27:11.091156 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260929 03:27:11.100876 8346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:11.100975 8346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:11.101594 8346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Sending DeleteTablet for 3 replicas of tablet 27fb5a58dac649ee9533e61e712ec225
I20260929 03:27:11.101778 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fb5a58dac649ee9533e61e712ec225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:56044
I20260929 03:27:11.101893 8948 tablet_replica.cc:330] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: stopping tablet replica
I20260929 03:27:11.101878 8770 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fb5a58dac649ee9533e61e712ec225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:57384
I20260929 03:27:11.101944 8948 raft_consensus.cc:2229] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:11.101955 8947 tablet_replica.cc:330] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: stopping tablet replica
I20260929 03:27:11.101999 8947 raft_consensus.cc:2229] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:11.102022 8948 raft_consensus.cc:2258] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.102039 8947 raft_consensus.cc:2258] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.102252 8628 tablet_service.cc:1514] Processing DeleteTablet for tablet 27fb5a58dac649ee9533e61e712ec225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:11 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53486
I20260929 03:27:11.102325 8947 ts_tablet_manager.cc:1905] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.102399 8948 ts_tablet_manager.cc:1905] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.103137 8949 tablet_replica.cc:330] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: stopping tablet replica
I20260929 03:27:11.103188 8949 raft_consensus.cc:2229] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:11.103216 8949 raft_consensus.cc:2258] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:11.103541 8949 ts_tablet_manager.cc:1905] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:11.103750 8948 ts_tablet_manager.cc:1918] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:11.103688 8346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:27:11.103798 8948 log.cc:1196] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/27fb5a58dac649ee9533e61e712ec225
W20260929 03:27:11.103838 8346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:11.103907 8947 ts_tablet_manager.cc:1918] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:11.103953 8947 log.cc:1196] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/27fb5a58dac649ee9533e61e712ec225
I20260929 03:27:11.104012 8948 ts_tablet_manager.cc:1939] T 27fb5a58dac649ee9533e61e712ec225 P 47da0c76896643d8a5727f56d334830c: Deleting consensus metadata
I20260929 03:27:11.104179 8947 ts_tablet_manager.cc:1939] T 27fb5a58dac649ee9533e61e712ec225 P 65d9153553674671b3b50cb00a4f2e83: Deleting consensus metadata
I20260929 03:27:11.104705 8333 catalog_manager.cc:4841] TS 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803): tablet 27fb5a58dac649ee9533e61e712ec225 (table test-table [id=720e9bf581c5449e87c3768be891027f]) successfully deleted
I20260929 03:27:11.104840 8334 catalog_manager.cc:4841] TS 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131): tablet 27fb5a58dac649ee9533e61e712ec225 (table test-table [id=720e9bf581c5449e87c3768be891027f]) successfully deleted
I20260929 03:27:11.105011 8485 tablet_service.cc:1467] Processing CreateTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 (DEFAULT_TABLE table=test-table [id=ead83dd2beae452da59921c6d74cc009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.105010 8770 tablet_service.cc:1467] Processing CreateTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 (DEFAULT_TABLE table=test-table [id=ead83dd2beae452da59921c6d74cc009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.105048 8628 tablet_service.cc:1467] Processing CreateTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 (DEFAULT_TABLE table=test-table [id=ead83dd2beae452da59921c6d74cc009]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:11.105105 8485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512fcbabedf04c6e9871c81f2bb11bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.105106 8770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512fcbabedf04c6e9871c81f2bb11bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.105173 8628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512fcbabedf04c6e9871c81f2bb11bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:11.106294 8958 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: Bootstrap starting.
I20260929 03:27:11.106310 8959 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Bootstrap starting.
I20260929 03:27:11.106326 8960 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Bootstrap starting.
I20260929 03:27:11.106586 8959 tablet_bootstrap.cc:654] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.106586 8960 tablet_bootstrap.cc:654] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.106614 8958 tablet_bootstrap.cc:654] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:11.107179 8960 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: No bootstrap required, opened a new log
I20260929 03:27:11.107230 8960 ts_tablet_manager.cc:1397] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:27:11.107275 8959 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: No bootstrap required, opened a new log
I20260929 03:27:11.107329 8959 ts_tablet_manager.cc:1397] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:11.107338 8960 raft_consensus.cc:348] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.107407 8960 raft_consensus.cc:374] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.107445 8960 raft_consensus.cc:729] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 068bf27fc2c44250aae286e14417dc6e, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.107439 8959 raft_consensus.cc:348] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.107504 8959 raft_consensus.cc:374] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.107487 8960 consensus_queue.cc:260] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.107532 8959 raft_consensus.cc:729] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d9153553674671b3b50cb00a4f2e83, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.107574 8960 ts_tablet_manager.cc:1428] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:11.107569 8959 consensus_queue.cc:260] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.107666 8959 ts_tablet_manager.cc:1428] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:11.105178 8949 ts_tablet_manager.cc:1918] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:11.107779 8949 log.cc:1196] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/27fb5a58dac649ee9533e61e712ec225
I20260929 03:27:11.107986 8949 ts_tablet_manager.cc:1939] T 27fb5a58dac649ee9533e61e712ec225 P 068bf27fc2c44250aae286e14417dc6e: Deleting consensus metadata
I20260929 03:27:11.108276 8333 catalog_manager.cc:4841] TS 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935): tablet 27fb5a58dac649ee9533e61e712ec225 (table test-table [id=720e9bf581c5449e87c3768be891027f]) successfully deleted
I20260929 03:27:11.114347 8958 tablet_bootstrap.cc:492] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: No bootstrap required, opened a new log
I20260929 03:27:11.114403 8958 ts_tablet_manager.cc:1397] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260929 03:27:11.114528 8958 raft_consensus.cc:348] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.114603 8958 raft_consensus.cc:374] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:11.114624 8958 raft_consensus.cc:729] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Initialized, Role: FOLLOWER
I20260929 03:27:11.114669 8958 consensus_queue.cc:260] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.114782 8958 ts_tablet_manager.cc:1428] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:11.145087 8979 raft_consensus.cc:482] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:11.145171 8979 raft_consensus.cc:504] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.145331 8979 leader_election.cc:288] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:11.145460 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512fcbabedf04c6e9871c81f2bb11bf3" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e" is_pre_election: true
I20260929 03:27:11.145473 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512fcbabedf04c6e9871c81f2bb11bf3" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83" is_pre_election: true
I20260929 03:27:11.145550 8649 raft_consensus.cc:2454] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 0.
I20260929 03:27:11.145563 8792 raft_consensus.cc:2454] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 0.
I20260929 03:27:11.145725 8439 leader_election.cc:302] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47da0c76896643d8a5727f56d334830c, 65d9153553674671b3b50cb00a4f2e83; no voters:
I20260929 03:27:11.145809 8979 raft_consensus.cc:2781] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:11.145884 8979 raft_consensus.cc:482] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:11.145913 8979 raft_consensus.cc:3037] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.146612 8979 raft_consensus.cc:504] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.146834 8979 leader_election.cc:288] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 election: Requested vote from peers 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935), 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131)
I20260929 03:27:11.146878 8649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512fcbabedf04c6e9871c81f2bb11bf3" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "068bf27fc2c44250aae286e14417dc6e"
I20260929 03:27:11.146943 8649 raft_consensus.cc:3037] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.146960 8792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512fcbabedf04c6e9871c81f2bb11bf3" candidate_uuid: "47da0c76896643d8a5727f56d334830c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d9153553674671b3b50cb00a4f2e83"
I20260929 03:27:11.147073 8792 raft_consensus.cc:3037] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:11.147599 8792 raft_consensus.cc:2454] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 1.
I20260929 03:27:11.147697 8649 raft_consensus.cc:2454] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47da0c76896643d8a5727f56d334830c in term 1.
I20260929 03:27:11.147765 8439 leader_election.cc:302] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47da0c76896643d8a5727f56d334830c, 65d9153553674671b3b50cb00a4f2e83; no voters:
I20260929 03:27:11.147848 8979 raft_consensus.cc:2781] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:11.147913 8979 raft_consensus.cc:686] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [term 1 LEADER]: Becoming Leader. State: Replica: 47da0c76896643d8a5727f56d334830c, State: Running, Role: LEADER
I20260929 03:27:11.147961 8979 consensus_queue.cc:237] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } }
I20260929 03:27:11.148465 8346 catalog_manager.cc:5495] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c reported cstate change: term changed from 0 to 1, leader changed from <none> to 47da0c76896643d8a5727f56d334830c (127.6.116.129). New cstate: current_term: 1 leader_uuid: "47da0c76896643d8a5727f56d334830c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47da0c76896643d8a5727f56d334830c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 35803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:11.155761 8792 raft_consensus.cc:1261] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Refusing update from remote peer 47da0c76896643d8a5727f56d334830c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:11.155774 8649 raft_consensus.cc:1261] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Refusing update from remote peer 47da0c76896643d8a5727f56d334830c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:11.156021 8979 consensus_queue.cc:1035] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d9153553674671b3b50cb00a4f2e83" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:11.156112 8964 consensus_queue.cc:1035] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c [LEADER]: Connected to new peer: Peer: permanent_uuid: "068bf27fc2c44250aae286e14417dc6e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:27:11.160328 8932 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 03:27:12.677729 8347 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:12.677832 8347 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:12.678673 8347 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5869f82e6ba4b358a9deeb1c7fa4d66: Sending DeleteTablet for 3 replicas of tablet 512fcbabedf04c6e9871c81f2bb11bf3
I20260929 03:27:12.678916 8485 tablet_service.cc:1514] Processing DeleteTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:12 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:56044
I20260929 03:27:12.678937 8770 tablet_service.cc:1514] Processing DeleteTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:12 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:57384
I20260929 03:27:12.679014 8628 tablet_service.cc:1514] Processing DeleteTablet for tablet 512fcbabedf04c6e9871c81f2bb11bf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:12 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53486
I20260929 03:27:12.679193 9010 tablet_replica.cc:330] T 512fcbabedf04c6e9871c81f2bb11bf3 P 47da0c76896643d8a5727f56d334830c: stopping tablet replica
I20260929 03:27:12.679284 9012 tablet_replica.cc:330] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: stopping tablet replica
I20260929 03:27:12.679292 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 8420
I20260929 03:27:12.680881 9011 tablet_replica.cc:330] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: stopping tablet replica
I20260929 03:27:12.680941 9012 raft_consensus.cc:2229] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:12.681016 9012 raft_consensus.cc:2258] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:12.681010 9011 raft_consensus.cc:2229] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:12.681061 9011 raft_consensus.cc:2258] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:12.681281 9012 ts_tablet_manager.cc:1905] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:12.681356 9011 ts_tablet_manager.cc:1905] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:12.683012 9012 ts_tablet_manager.cc:1918] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:27:12.683146 9012 log.cc:1196] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/512fcbabedf04c6e9871c81f2bb11bf3
I20260929 03:27:12.683369 9012 ts_tablet_manager.cc:1939] T 512fcbabedf04c6e9871c81f2bb11bf3 P 068bf27fc2c44250aae286e14417dc6e: Deleting consensus metadata
I20260929 03:27:12.683374 9011 ts_tablet_manager.cc:1918] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 03:27:12.683434 9011 log.cc:1196] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/512fcbabedf04c6e9871c81f2bb11bf3
I20260929 03:27:12.683686 9011 ts_tablet_manager.cc:1939] T 512fcbabedf04c6e9871c81f2bb11bf3 P 65d9153553674671b3b50cb00a4f2e83: Deleting consensus metadata
I20260929 03:27:12.683698 8333 catalog_manager.cc:4841] TS 068bf27fc2c44250aae286e14417dc6e (127.6.116.130:45935): tablet 512fcbabedf04c6e9871c81f2bb11bf3 (table test-table [id=ead83dd2beae452da59921c6d74cc009]) successfully deleted
I20260929 03:27:12.684054 8334 catalog_manager.cc:4841] TS 65d9153553674671b3b50cb00a4f2e83 (127.6.116.131:36131): tablet 512fcbabedf04c6e9871c81f2bb11bf3 (table test-table [id=ead83dd2beae452da59921c6d74cc009]) successfully deleted
W20260929 03:27:12.686337 8333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:35803 (error 108)
I20260929 03:27:12.686510 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 8563
I20260929 03:27:12.687510 9014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:12.687282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:35803 (local address 127.0.0.1:56054)
0929 03:27:12.687412 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:12.687464 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:35803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
W20260929 03:27:12.687654 8333 catalog_manager.cc:4584] TS 47da0c76896643d8a5727f56d334830c (127.6.116.129:35803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 512fcbabedf04c6e9871c81f2bb11bf3: Network error: Client connection negotiation failed: client connection to 127.6.116.129:35803: connect: Connection refused (error 111)
I20260929 03:27:12.693470 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 8704
I20260929 03:27:12.698499 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 8315
2026-09-29T03:27:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5859 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:12 dist-test-slave-z2zg krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:12 dist-test-slave-z2zg krb5kdc[9017](info): set up 2 sockets
Sep 29 03:27:12 dist-test-slave-z2zg krb5kdc[9017](info): commencing operation
krb5kdc: starting...
W20260929 03:27:13.728811 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:13 dist-test-slave-z2zg krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:13.837649 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40651
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:40651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:13.914624 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:13.914808 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:13.914829 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:13.915938 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:13.915978 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:13.915992 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:13.916008 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:13.916020 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:13.917380 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:40651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:13.917555 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:13.917716 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:13.922169 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:13.922178 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:13.922431 9030 server_base.cc:1034] running on GCE node
W20260929 03:27:13.922188 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:13.922711 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:13.922987 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:13.924155 9030 hybrid_clock.cc:648] HybridClock initialized: now 1790652433924142 us; error 27 us; skew 500 ppm
Sep 29 03:27:13 dist-test-slave-z2zg krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652433, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:13.926600 9030 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:13.926994 9030 webserver.cc:460] Webserver started at http://127.6.116.190:43099/ using document root <none> and password file <none>
I20260929 03:27:13.927248 9030 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:13.927315 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:13.927461 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:13.928422 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "f2315d69a6354fd3bfdc48555c0e811d"
format_stamp: "Formatted at 2026-09-29 03:27:13 on dist-test-slave-z2zg"
server_key: "97421af3b444dbea313deb8442908436"
server_key_iv: "f940da0963390987061203fa6f004504"
server_key_version: "encryptionkey@0"
I20260929 03:27:13.928754 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "f2315d69a6354fd3bfdc48555c0e811d"
format_stamp: "Formatted at 2026-09-29 03:27:13 on dist-test-slave-z2zg"
server_key: "97421af3b444dbea313deb8442908436"
server_key_iv: "f940da0963390987061203fa6f004504"
server_key_version: "encryptionkey@0"
I20260929 03:27:13.930038 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:13.930755 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:13.930920 9030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:13.931054 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "f2315d69a6354fd3bfdc48555c0e811d"
format_stamp: "Formatted at 2026-09-29 03:27:13 on dist-test-slave-z2zg"
server_key: "97421af3b444dbea313deb8442908436"
server_key_iv: "f940da0963390987061203fa6f004504"
server_key_version: "encryptionkey@0"
I20260929 03:27:13.931143 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:13.947023 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:13.949301 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:13.949455 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:13.953253 9030 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:40651
I20260929 03:27:13.953299 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:40651 every 8 connection(s)
I20260929 03:27:13.953624 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:13.954432 9098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:13.956573 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d: Bootstrap starting.
I20260929 03:27:13.957109 9098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:13.957389 9098 log.cc:824] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:13.958067 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d: No bootstrap required, opened a new log
I20260929 03:27:13.959648 9098 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } }
I20260929 03:27:13.959770 9098 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:13.959791 9098 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2315d69a6354fd3bfdc48555c0e811d, State: Initialized, Role: FOLLOWER
I20260929 03:27:13.959945 9098 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } }
I20260929 03:27:13.960003 9098 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:13.960023 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:13.960090 9098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:13.960551 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } }
I20260929 03:27:13.960659 9098 leader_election.cc:302] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2315d69a6354fd3bfdc48555c0e811d; no voters:
I20260929 03:27:13.960860 9098 leader_election.cc:288] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:13.960939 9101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:13.961109 9098 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:13.961309 9101 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [term 1 LEADER]: Becoming Leader. State: Replica: f2315d69a6354fd3bfdc48555c0e811d, State: Running, Role: LEADER
I20260929 03:27:13.961407 9101 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } }
I20260929 03:27:13.961603 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9030
I20260929 03:27:13.961702 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:13.961898 6610 external_mini_cluster.cc:1428] Setting key bd6830d99e6ef1c01b17c1ae68baae1c
I20260929 03:27:13.962062 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2315d69a6354fd3bfdc48555c0e811d. Latest consensus state: current_term: 1 leader_uuid: "f2315d69a6354fd3bfdc48555c0e811d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } } }
I20260929 03:27:13.962208 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:13.962236 9102 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2315d69a6354fd3bfdc48555c0e811d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2315d69a6354fd3bfdc48555c0e811d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40651 } } }
I20260929 03:27:13.962303 9102 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:13.962699 9114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:13.963361 9114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:13.964694 9114 catalog_manager.cc:1347] Generated new cluster ID: 801da538714747f2bcffcf7bfb412c66
I20260929 03:27:13.964745 9114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:13 dist-test-slave-z2zg krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:13.973820 9114 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:13.974308 9114 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:13.975965 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:13.962595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.6.116.190:40651)
0929 03:27:13.962831 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:13.962839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:13.962889 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:27:13.962999 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:13.963006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:13.963061 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:13.963249 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:13.964126 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:13.967958 (+ 3832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:13.969082 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:13.969652 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:13.969744 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:13.972424 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:13.972427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:13.972428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:13.972449 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:13.974348 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:13.974774 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:13.974778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:13.974778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:13.974883 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:13.975098 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:13.975099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:13.975099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:13.975371 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:13.975509 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:13.975618 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:27:13.975751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260929 03:27:13.988248 9114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2315d69a6354fd3bfdc48555c0e811d: Generated new TSK 0
I20260929 03:27:13.988411 9114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:14.015614 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:14.095424 9124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:14.095625 9124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:14.095666 9124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:14.096771 9124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:14.096829 9124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:14.096871 9124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:14.098187 9124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:14.098421 9124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:14.098640 9124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:14.103449 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:14.103468 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:14.103468 9132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:14.103672 9124 server_base.cc:1034] running on GCE node
I20260929 03:27:14.103874 9124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:14.104137 9124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:14.105304 9124 hybrid_clock.cc:648] HybridClock initialized: now 1790652434105290 us; error 24 us; skew 500 ppm
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:14.107710 9124 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:14.108124 9124 webserver.cc:460] Webserver started at http://127.6.116.129:35697/ using document root <none> and password file <none>
I20260929 03:27:14.108345 9124 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:14.108409 9124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:14.108572 9124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:14.109537 9124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "317bc8ed9a6246c8bead8bb42cae3fc1"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "8c60b459826d6301a9a53e01ea43f681"
server_key_iv: "e2bd5d8113023549808382d724f1bbcd"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.109942 9124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "317bc8ed9a6246c8bead8bb42cae3fc1"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "8c60b459826d6301a9a53e01ea43f681"
server_key_iv: "e2bd5d8113023549808382d724f1bbcd"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.111229 9124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:27:14.111887 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.112051 9124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.112136 9124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "317bc8ed9a6246c8bead8bb42cae3fc1"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "8c60b459826d6301a9a53e01ea43f681"
server_key_iv: "e2bd5d8113023549808382d724f1bbcd"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.112217 9124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:14.141263 9124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:14.142939 9124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:14.143129 9124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:14.143343 9124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:14.143664 9124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:14.143718 9124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.143760 9124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:14.143788 9124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.149340 9124 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:45541
I20260929 03:27:14.149415 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:45541 every 8 connection(s)
I20260929 03:27:14.149699 9124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:14.150195 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9124
I20260929 03:27:14.150292 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:14.150480 6610 external_mini_cluster.cc:1428] Setting key a64a9e73a847492b838f142bc069dcab
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:14.160283 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.150365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43231 (local address 127.6.116.190:40651)
0929 03:27:14.150416 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:14.150418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:14.150645 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:27:14.151302 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:14.151302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:14.151312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:14.151350 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:14.152570 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.153385 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.154729 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.154925 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.154980 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.157303 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:14.157304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:14.157304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:14.157312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:14.158763 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.159275 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.159276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.159276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.159337 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.159613 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.159614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.159614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.159727 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:14.159831 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:14.160088 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 03:27:14.160189 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:14.160409 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.150362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40651 (local address 127.6.116.129:43231)
0929 03:27:14.150519 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:14.150564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:14.151182 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:14.151422 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:14.151426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:14.152215 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:14.152475 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.152482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.153471 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.153472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.154635 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.154638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.155017 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.155018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.155183 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.155693 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:14.155703 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:14.157207 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:14.158863 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.158870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.158872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.159188 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.159402 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.159403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.159404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.159532 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.159800 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:14.159808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:14.159949 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 03:27:14.160047 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 03:27:14.160198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260929 03:27:14.160861 9253 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40651
I20260929 03:27:14.160948 9253 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:14.161144 9253 heartbeater.cc:511] Master 127.6.116.190:40651 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
I20260929 03:27:14.161650 9062 ts_manager.cc:194] Registered new tserver with Master: 317bc8ed9a6246c8bead8bb42cae3fc1 (127.6.116.129:45541)
I20260929 03:27:14.162633 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:43231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:14.175225 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:14.252422 9258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:14.252600 9258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:14.252624 9258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:14.253823 9258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:14.253866 9258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:14.253893 9258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:14.255246 9258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:14.255409 9258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:14.255558 9258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:14.260104 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:14.260108 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:14.260120 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:14.260399 9258 server_base.cc:1034] running on GCE node
I20260929 03:27:14.260564 9258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:14.260754 9258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:14.261947 9258 hybrid_clock.cc:648] HybridClock initialized: now 1790652434261933 us; error 26 us; skew 500 ppm
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:14.264428 9258 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:14.264880 9258 webserver.cc:460] Webserver started at http://127.6.116.130:46027/ using document root <none> and password file <none>
I20260929 03:27:14.265120 9258 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:14.265162 9258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:14.265295 9258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:14.266274 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "6fa7e7565fc74f3db96b1aa870c0f741"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "5f394b842383cbfd84a128905dcbdd21"
server_key_iv: "e0cbcc6ff35ccb2c4356f40d75ccefe6"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.266635 9258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "6fa7e7565fc74f3db96b1aa870c0f741"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "5f394b842383cbfd84a128905dcbdd21"
server_key_iv: "e0cbcc6ff35ccb2c4356f40d75ccefe6"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.267962 9258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:14.268602 9273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.268786 9258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:14.268877 9258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "6fa7e7565fc74f3db96b1aa870c0f741"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "5f394b842383cbfd84a128905dcbdd21"
server_key_iv: "e0cbcc6ff35ccb2c4356f40d75ccefe6"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.268939 9258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:14.291283 9258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:14.292884 9258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:14.293025 9258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:14.293215 9258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:14.293479 9258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:14.293529 9258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.293565 9258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:14.293591 9258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.299012 9258 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:40265
I20260929 03:27:14.299090 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:40265 every 8 connection(s)
I20260929 03:27:14.299372 9258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:14.307446 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.300026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55289 (local address 127.6.116.190:40651)
0929 03:27:14.300097 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:14.300099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:14.300270 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:27:14.300700 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:14.300701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:14.300712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:14.300751 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:14.301520 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.302020 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.302896 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.303086 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.303147 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.304923 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:14.304925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:14.304925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:14.304931 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:14.306336 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.306652 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.306652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.306653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.306700 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.306900 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.306900 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.306901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.306989 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:14.307036 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:14.307326 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 03:27:14.307386 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:27:14.307612 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.299981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40651 (local address 127.6.116.130:55289)
0929 03:27:14.300147 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:14.300190 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:14.300610 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:14.300797 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:14.300800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:14.301285 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:14.301448 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.301452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.302095 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.302096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.302824 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.302825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.303153 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.303154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.303242 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.303595 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:14.303602 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:14.304845 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:14.306402 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.306406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.306408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.306591 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.306766 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.306767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.306767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.306849 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.307095 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:14.307100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:14.307218 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 03:27:14.307298 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:27:14.307402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260929 03:27:14.308070 9387 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40651
I20260929 03:27:14.308163 9387 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:14.308362 9387 heartbeater.cc:511] Master 127.6.116.190:40651 requested a full tablet report, sending...
I20260929 03:27:14.308717 9062 ts_manager.cc:194] Registered new tserver with Master: 6fa7e7565fc74f3db96b1aa870c0f741 (127.6.116.130:40265)
I20260929 03:27:14.309060 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:55289
I20260929 03:27:14.309356 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9258
I20260929 03:27:14.309448 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:14.309625 6610 external_mini_cluster.cc:1428] Setting key 751361ae09a9e1d7ae8b02ba77e1f70b
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:14.333482 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:14.414788 9392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:14.415015 9392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:14.415040 9392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:14.416262 9392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:14.416302 9392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:14.416328 9392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:14.417734 9392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:14.417964 9392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:14.418183 9392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:14.422902 9398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:14.422895 9397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:14.423138 9392 server_base.cc:1034] running on GCE node
W20260929 03:27:14.422896 9400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:14.423390 9392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:14.423633 9392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:14.424798 9392 hybrid_clock.cc:648] HybridClock initialized: now 1790652434424792 us; error 36 us; skew 500 ppm
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:14.427296 9392 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:14.427736 9392 webserver.cc:460] Webserver started at http://127.6.116.131:43631/ using document root <none> and password file <none>
I20260929 03:27:14.427961 9392 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:14.428027 9392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:14.428166 9392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:14.429208 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "8cab547945ab491bad109a73da4cec89"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "f867457eb142736daaca2d6d3a085719"
server_key_iv: "986c8248f333f4af2127978405daf76e"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.429554 9392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "8cab547945ab491bad109a73da4cec89"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "f867457eb142736daaca2d6d3a085719"
server_key_iv: "986c8248f333f4af2127978405daf76e"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.430848 9392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:14.431632 9407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.431805 9392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.431874 9392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "8cab547945ab491bad109a73da4cec89"
format_stamp: "Formatted at 2026-09-29 03:27:14 on dist-test-slave-z2zg"
server_key: "f867457eb142736daaca2d6d3a085719"
server_key_iv: "986c8248f333f4af2127978405daf76e"
server_key_version: "encryptionkey@0"
I20260929 03:27:14.431962 9392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:14.446719 9392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:14.448619 9392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:14.448773 9392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:14.448987 9392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:14.449282 9392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:14.449337 9392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.449378 9392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:14.449405 9392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:14.454794 9392 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:44313
I20260929 03:27:14.454864 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:44313 every 8 connection(s)
I20260929 03:27:14.455183 9392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:14.459174 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9392
I20260929 03:27:14.459285 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:14.459486 6610 external_mini_cluster.cc:1428] Setting key d24d6f549b68594780e0074710227d33
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:14.464211 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.455870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:49777 (local address 127.6.116.190:40651)
0929 03:27:14.455919 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:14.455922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:14.456129 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:27:14.456552 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:14.456553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:14.456564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:14.456618 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:14.457462 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.458013 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.458914 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.459175 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.459227 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.461604 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:14.461606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:14.461606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:14.461612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:14.463114 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.463464 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.463464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.463465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.463513 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:14.463730 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:14.463730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:14.463731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:14.463824 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:14.463872 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:14.464091 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 03:27:14.464151 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:27:14.464386 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:14.455852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40651 (local address 127.6.116.131:49777)
0929 03:27:14.456014 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:14.456045 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:14.456474 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:14.456690 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:14.456693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:14.457211 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:14.457377 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.457381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.458095 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.458096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.458842 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:14.458844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:14.459254 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:14.459255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:14.459404 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:14.459894 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:14.459903 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:14.461508 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:14.463192 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.463197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.463199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.463399 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.463584 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:14.463585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:14.463585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:14.463676 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:14.463876 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:14.463880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:14.463982 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 03:27:14.464065 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:27:14.464173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260929 03:27:14.464793 9522 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40651
I20260929 03:27:14.464884 9522 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:14.465045 9522 heartbeater.cc:511] Master 127.6.116.190:40651 requested a full tablet report, sending...
I20260929 03:27:14.465420 9062 ts_manager.cc:194] Registered new tserver with Master: 8cab547945ab491bad109a73da4cec89 (127.6.116.131:44313)
I20260929 03:27:14.465797 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:49777
I20260929 03:27:14.471334 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:27:14.482426 9118 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:14.481519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.6.116.190:40651)
0929 03:27:14.481604 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:14.481607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:14.481712 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:27:14.481789 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:14.481790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:14.481806 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:14.481868 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:14.482363 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54948: BlockingRecv error: recv got EOF from 127.0.0.1:54948 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 03:27:14.483266 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9124
I20260929 03:27:14.487704 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9258
I20260929 03:27:14.491876 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9392
I20260929 03:27:14.495941 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9030
2026-09-29T03:27:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1796 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9534](info): set up 2 sockets
Sep 29 03:27:14 dist-test-slave-z2zg krb5kdc[9534](info): commencing operation
krb5kdc: starting...
W20260929 03:27:15.512578 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:15.625891 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43777
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:15.705107 9547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:15.705351 9547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:15.705390 9547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:15.706598 9547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:15.706674 9547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:15.706702 9547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:15.706736 9547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:15.706772 9547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:15.708256 9547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:15.708542 9547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:15.708765 9547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:15.713531 9555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:15.713536 9553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:15.713634 9547 server_base.cc:1034] running on GCE node
W20260929 03:27:15.713546 9552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:15.714105 9547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:15.714331 9547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:15.715463 9547 hybrid_clock.cc:648] HybridClock initialized: now 1790652435715454 us; error 31 us; skew 500 ppm
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:15.717923 9547 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:15.718381 9547 webserver.cc:460] Webserver started at http://127.6.116.190:46101/ using document root <none> and password file <none>
I20260929 03:27:15.718602 9547 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:15.718647 9547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:15.718806 9547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:15.719882 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "4b7318206d6c4055a57ed9f66fc11842"
format_stamp: "Formatted at 2026-09-29 03:27:15 on dist-test-slave-z2zg"
server_key: "e6ae85a38c0e128b88b000f1b66da54a"
server_key_iv: "686c15efbb2f2ccc7974b1136beddd1e"
server_key_version: "encryptionkey@0"
I20260929 03:27:15.720235 9547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "4b7318206d6c4055a57ed9f66fc11842"
format_stamp: "Formatted at 2026-09-29 03:27:15 on dist-test-slave-z2zg"
server_key: "e6ae85a38c0e128b88b000f1b66da54a"
server_key_iv: "686c15efbb2f2ccc7974b1136beddd1e"
server_key_version: "encryptionkey@0"
I20260929 03:27:15.721531 9547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:15.722281 9562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:15.722500 9547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:15.722601 9547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "4b7318206d6c4055a57ed9f66fc11842"
format_stamp: "Formatted at 2026-09-29 03:27:15 on dist-test-slave-z2zg"
server_key: "e6ae85a38c0e128b88b000f1b66da54a"
server_key_iv: "686c15efbb2f2ccc7974b1136beddd1e"
server_key_version: "encryptionkey@0"
I20260929 03:27:15.722653 9547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:15.735584 9547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:15.737201 9547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:15.737308 9547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:15.741331 9547 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:43777
I20260929 03:27:15.741379 9614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:43777 every 8 connection(s)
I20260929 03:27:15.741693 9547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:15.742552 9615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:15.744685 9615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: Bootstrap starting.
I20260929 03:27:15.745194 9615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:15.745424 9615 log.cc:824] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:15.746079 9615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: No bootstrap required, opened a new log
I20260929 03:27:15.747884 9615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } }
I20260929 03:27:15.748028 9615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:15.748055 9615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7318206d6c4055a57ed9f66fc11842, State: Initialized, Role: FOLLOWER
I20260929 03:27:15.748188 9615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } }
I20260929 03:27:15.748268 9615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:15.748294 9615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:15.748323 9615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:15.749105 9615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } }
I20260929 03:27:15.749192 9615 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b7318206d6c4055a57ed9f66fc11842; no voters:
I20260929 03:27:15.749315 9615 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:15.749366 9618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:15.749562 9618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7318206d6c4055a57ed9f66fc11842, State: Running, Role: LEADER
I20260929 03:27:15.749593 9615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:15.749651 9618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } }
I20260929 03:27:15.750140 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9547
I20260929 03:27:15.750259 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:15.750445 6610 external_mini_cluster.cc:1428] Setting key cc84af89a62438a1a29a2adb9c478f60
I20260929 03:27:15.750950 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7318206d6c4055a57ed9f66fc11842. Latest consensus state: current_term: 1 leader_uuid: "4b7318206d6c4055a57ed9f66fc11842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } } }
I20260929 03:27:15.750955 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b7318206d6c4055a57ed9f66fc11842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b7318206d6c4055a57ed9f66fc11842" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43777 } } }
I20260929 03:27:15.751346 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:15.751358 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842 [sys.catalog]: This master's current role is: LEADER
W20260929 03:27:15.751536 9634 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:15.751595 9634 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:15.751682 9637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:15.751870 9637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:15.753230 9637 catalog_manager.cc:1347] Generated new cluster ID: 58be7426377d4befaed6258c034d19a4
I20260929 03:27:15.753280 9637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:15.758878 9637 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:15.759529 9637 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:15.763638 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:15.751139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.6.116.190:43777)
0929 03:27:15.751296 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:15.751305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:15.751581 (+ 276us) server_negotiation.cc:407] Connection header received
0929 03:27:15.751755 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:15.751762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:15.751820 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:15.751974 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:15.752851 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:15.754192 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:15.755242 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:15.755847 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:15.755976 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:15.758720 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:15.758723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:15.758724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:15.758736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:15.761534 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:15.761970 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:15.761972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:15.761973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:15.762102 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:15.762329 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:15.762330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:15.762330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:15.762602 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:15.763219 (+ 617us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:15.763293 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:27:15.763435 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260929 03:27:15.764587 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:27:15.772230 9637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7318206d6c4055a57ed9f66fc11842: Generated new TSK 0
I20260929 03:27:15.772389 9637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:15.779824 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:15.776205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.6.116.190:43777)
0929 03:27:15.776269 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:15.776271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:15.776421 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:27:15.776452 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:15.776453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:15.776465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:15.776529 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:15.776862 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:15.777585 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:15.778304 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:15.778610 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:15.778689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:15.779327 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:15.779328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:15.779329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:15.779547 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:15.779564 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:15.779571 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:15.779640 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:15.779724 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 03:27:15.779750 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:27:15.782444 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9547
2026-09-29T03:27:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1284 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9648](info): set up 2 sockets
Sep 29 03:27:15 dist-test-slave-z2zg krb5kdc[9648](info): commencing operation
krb5kdc: starting...
W20260929 03:27:16.800554 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:16 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:16.909224 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45177
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:16.986737 9661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:16.986930 9661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:16.986950 9661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:16.988026 9661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:16.988066 9661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:16.988080 9661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:16.988092 9661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:16.988103 9661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:16.989419 9661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:45177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:16.989598 9661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:16.989742 9661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:16.994248 9667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:16.994274 9669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:16.994400 9661 server_base.cc:1034] running on GCE node
W20260929 03:27:16.994266 9666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:16.994661 9661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:16.994946 9661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:16.996105 9661 hybrid_clock.cc:648] HybridClock initialized: now 1790652436996084 us; error 50 us; skew 500 ppm
Sep 29 03:27:16 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652436, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:16.998288 9661 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:16.998741 9661 webserver.cc:460] Webserver started at http://127.6.116.190:45821/ using document root <none> and password file <none>
I20260929 03:27:16.998945 9661 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:16.999006 9661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:16.999176 9661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:17.000110 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "1ce76d33c99b46afb5c5a930e3d739cf"
format_stamp: "Formatted at 2026-09-29 03:27:16 on dist-test-slave-z2zg"
server_key: "33d313ff352479af205a0e13385f1b9a"
server_key_iv: "4e47c17c853ff2996a2d945ae100f268"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.000453 9661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "1ce76d33c99b46afb5c5a930e3d739cf"
format_stamp: "Formatted at 2026-09-29 03:27:16 on dist-test-slave-z2zg"
server_key: "33d313ff352479af205a0e13385f1b9a"
server_key_iv: "4e47c17c853ff2996a2d945ae100f268"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.001696 9661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:17.002460 9676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.002621 9661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:17.002730 9661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "1ce76d33c99b46afb5c5a930e3d739cf"
format_stamp: "Formatted at 2026-09-29 03:27:16 on dist-test-slave-z2zg"
server_key: "33d313ff352479af205a0e13385f1b9a"
server_key_iv: "4e47c17c853ff2996a2d945ae100f268"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.002823 9661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:17.014767 9661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:17.016435 9661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:17.016536 9661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:17.020292 9661 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:45177
I20260929 03:27:17.020351 9728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:45177 every 8 connection(s)
I20260929 03:27:17.020668 9661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:17.021538 9729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:17.023191 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9661
I20260929 03:27:17.023276 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:17.023468 6610 external_mini_cluster.cc:1428] Setting key 19f939d51f0e53850a702439127531b0
I20260929 03:27:17.023953 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: Bootstrap starting.
I20260929 03:27:17.024549 9729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:17.024840 9729 log.cc:824] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:17.025501 9729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: No bootstrap required, opened a new log
I20260929 03:27:17.027333 9729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } }
I20260929 03:27:17.027472 9729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:17.027515 9729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce76d33c99b46afb5c5a930e3d739cf, State: Initialized, Role: FOLLOWER
I20260929 03:27:17.027633 9729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } }
I20260929 03:27:17.027738 9729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:17.027797 9729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:17.027846 9729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:17.028486 9729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } }
I20260929 03:27:17.028599 9729 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce76d33c99b46afb5c5a930e3d739cf; no voters:
I20260929 03:27:17.028769 9729 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:17.028786 9734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:17.028895 9734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [term 1 LEADER]: Becoming Leader. State: Replica: 1ce76d33c99b46afb5c5a930e3d739cf, State: Running, Role: LEADER
I20260929 03:27:17.029009 9729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:17.028993 9734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } }
I20260929 03:27:17.030962 9735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } } }
I20260929 03:27:17.031020 9735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:17.030824 9736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce76d33c99b46afb5c5a930e3d739cf. Latest consensus state: current_term: 1 leader_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce76d33c99b46afb5c5a930e3d739cf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45177 } } }
I20260929 03:27:17.031279 9736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf [sys.catalog]: This master's current role is: LEADER
W20260929 03:27:17.031313 9748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:17.031422 9748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:17.031497 9750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:17.031661 9750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:17.032961 9750 catalog_manager.cc:1347] Generated new cluster ID: b962b36cf4ef4773959eee6a05b59369
I20260929 03:27:17.033022 9750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:17.034138 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.023931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.6.116.190:45177)
0929 03:27:17.024092 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:17.024100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:17.024123 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:27:17.024525 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:17.024531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:17.024573 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:17.024751 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:17.025445 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.026368 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.027205 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.027679 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.027801 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.030538 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:17.030541 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:17.030542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:17.030550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:17.032547 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.032956 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.032959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.032959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.033084 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.033295 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.033295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.033296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.033557 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:17.033684 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:17.033834 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:27:17.033952 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260929 03:27:17.050854 9750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:17.051563 9750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:17.057835 9750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce76d33c99b46afb5c5a930e3d739cf: Generated new TSK 0
I20260929 03:27:17.058009 9750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:17.085309 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:17.165014 9755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:17.165251 9755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:17.165289 9755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:17.166446 9755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:17.166517 9755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:17.166565 9755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:17.168043 9755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:17.168303 9755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:17.168546 9755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:17.173381 9763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:17.173422 9760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:17.173419 9761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:17.173605 9755 server_base.cc:1034] running on GCE node
I20260929 03:27:17.173859 9755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:17.174060 9755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:17.175197 9755 hybrid_clock.cc:648] HybridClock initialized: now 1790652437175197 us; error 36 us; skew 500 ppm
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:17.177743 9755 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:17.178184 9755 webserver.cc:460] Webserver started at http://127.6.116.129:38223/ using document root <none> and password file <none>
I20260929 03:27:17.178388 9755 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:17.178452 9755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:17.178614 9755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:17.179621 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "0963a9ab57344c7d824d9a299a6fbc6f"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "dc54370e493059e24c2dcf741567abe1"
server_key_iv: "948016623361ae50277f6b507aae4342"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.179975 9755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "0963a9ab57344c7d824d9a299a6fbc6f"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "dc54370e493059e24c2dcf741567abe1"
server_key_iv: "948016623361ae50277f6b507aae4342"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.181236 9755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:17.181921 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.182086 9755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:17.182179 9755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "0963a9ab57344c7d824d9a299a6fbc6f"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "dc54370e493059e24c2dcf741567abe1"
server_key_iv: "948016623361ae50277f6b507aae4342"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.182258 9755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:17.198093 9755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:17.199899 9755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:17.200047 9755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:17.200244 9755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:17.200623 9755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:17.200675 9755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.200703 9755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:17.200729 9755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.206733 9755 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:34973
I20260929 03:27:17.206806 9883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:34973 every 8 connection(s)
I20260929 03:27:17.207135 9755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:17.209738 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9755
I20260929 03:27:17.209831 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:17.210022 6610 external_mini_cluster.cc:1428] Setting key f67e1d24631a73c86607e55e3f4d81cb
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:17.216836 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.207798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:44417 (local address 127.6.116.190:45177)
0929 03:27:17.207856 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:17.207858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:17.208028 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:27:17.208511 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:17.208512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:17.208524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:17.208576 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:17.209478 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.210274 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.211705 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.211932 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.212011 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.214026 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:17.214027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:17.214028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:17.214035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:17.215489 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.215891 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.215892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.215893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.215951 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.216199 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.216200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.216201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.216316 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:17.216387 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:17.216666 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:27:17.216751 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:17.217036 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.207768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45177 (local address 127.6.116.129:44417)
0929 03:27:17.207917 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:17.207949 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:17.208412 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:17.208669 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:17.208672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:17.209197 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:17.209371 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.209375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.210387 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.210388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.211628 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.211631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.212030 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.212031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.212169 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.212663 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:17.212671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:17.213930 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:17.215584 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.215591 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.215593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.215803 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.216017 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.216018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.216018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.216131 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.216392 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:17.216398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:17.216524 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 03:27:17.216628 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:27:17.216773 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260929 03:27:17.217437 9884 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45177
I20260929 03:27:17.217519 9884 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:17.217691 9884 heartbeater.cc:511] Master 127.6.116.190:45177 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:17.218194 9693 ts_manager.cc:194] Registered new tserver with Master: 0963a9ab57344c7d824d9a299a6fbc6f (127.6.116.129:34973)
I20260929 03:27:17.218811 9693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:44417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:17.233600 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:17.311054 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:17.311270 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:17.311311 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:17.312407 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:17.312470 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:17.312512 9889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:17.313872 9889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:17.314113 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:17.314352 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:17.319065 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:17.319077 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:17.319156 9889 server_base.cc:1034] running on GCE node
W20260929 03:27:17.319078 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:17.319510 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:17.319741 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:17.320914 9889 hybrid_clock.cc:648] HybridClock initialized: now 1790652437320895 us; error 32 us; skew 500 ppm
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:17.323230 9889 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:17.323632 9889 webserver.cc:460] Webserver started at http://127.6.116.130:34591/ using document root <none> and password file <none>
I20260929 03:27:17.323843 9889 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:17.323908 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:17.324057 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:17.324999 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "6b3ec2807e5a49ee89707ef1dc30d717"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "1e2c472086afa37e7440fd58af6887ca"
server_key_iv: "90a81e23c0c878e36cf89b347e255fbc"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.325356 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "6b3ec2807e5a49ee89707ef1dc30d717"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "1e2c472086afa37e7440fd58af6887ca"
server_key_iv: "90a81e23c0c878e36cf89b347e255fbc"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.326596 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:27:17.327348 9904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.327521 9889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:17.327616 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "6b3ec2807e5a49ee89707ef1dc30d717"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "1e2c472086afa37e7440fd58af6887ca"
server_key_iv: "90a81e23c0c878e36cf89b347e255fbc"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.327713 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:17.348272 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:17.349947 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:17.350098 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:17.350306 9889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:17.350589 9889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:17.350690 9889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.350754 9889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:17.350786 9889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.356297 9889 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:33133
I20260929 03:27:17.356371 10017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:33133 every 8 connection(s)
I20260929 03:27:17.356701 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:17.357672 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 9889
I20260929 03:27:17.357764 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:17.357908 6610 external_mini_cluster.cc:1428] Setting key 34066d0aac8589545e6ad7728542ade0
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:27:17.366664 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.357334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:35055 (local address 127.6.116.190:45177)
0929 03:27:17.357411 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:17.357413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:17.357620 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:27:17.358209 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:17.358210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:17.358221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:17.358290 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:17.359279 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.360055 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.361396 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.361728 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.361802 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.363848 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:17.363849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:17.363849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:17.363858 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:17.365431 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.365795 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.365796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.365796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.365858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.366071 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.366072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.366073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.366215 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:17.366295 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:17.366526 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 03:27:17.366585 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:27:17.366775 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.357299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45177 (local address 127.6.116.130:35055)
0929 03:27:17.357491 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:17.357523 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:17.358103 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:17.358340 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:17.358344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:17.358956 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:17.359186 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.359190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.360177 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.360178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.361313 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.361315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.361811 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.361812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.361900 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.362387 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:17.362398 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:17.363751 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:17.365506 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.365511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.365512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.365722 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.365923 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.365924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.365924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.366008 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.366301 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:17.366306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:17.366405 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 03:27:17.366479 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:27:17.366583 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260929 03:27:17.367450 10018 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45177
I20260929 03:27:17.367537 10018 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:17.367717 10018 heartbeater.cc:511] Master 127.6.116.190:45177 requested a full tablet report, sending...
I20260929 03:27:17.368074 9693 ts_manager.cc:194] Registered new tserver with Master: 6b3ec2807e5a49ee89707ef1dc30d717 (127.6.116.130:33133)
I20260929 03:27:17.368420 9693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:35055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:17.381845 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:17.459173 10023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:17.459360 10023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:17.459383 10023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:17.460464 10023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:17.460500 10023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:17.460525 10023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:17.462015 10023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:17.462199 10023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:17.462361 10023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:17.467032 10029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:17.467027 10028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:17.467211 10023 server_base.cc:1034] running on GCE node
W20260929 03:27:17.467082 10031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:17.467504 10023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:17.467736 10023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:17.468892 10023 hybrid_clock.cc:648] HybridClock initialized: now 1790652437468888 us; error 35 us; skew 500 ppm
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:17.471302 10023 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:17.471657 10023 webserver.cc:460] Webserver started at http://127.6.116.131:46129/ using document root <none> and password file <none>
I20260929 03:27:17.471895 10023 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:17.471938 10023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:17.472085 10023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:17.473032 10023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "032687e7e3f445a48f43cfeed1458d79"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "a71704a52537c2ebea68783cc2d272fa"
server_key_iv: "a3728c142deb718c859b33f134ba58db"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.473376 10023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "032687e7e3f445a48f43cfeed1458d79"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "a71704a52537c2ebea68783cc2d272fa"
server_key_iv: "a3728c142deb718c859b33f134ba58db"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.474596 10023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:17.475322 10038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.475507 10023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:17.475598 10023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "032687e7e3f445a48f43cfeed1458d79"
format_stamp: "Formatted at 2026-09-29 03:27:17 on dist-test-slave-z2zg"
server_key: "a71704a52537c2ebea68783cc2d272fa"
server_key_iv: "a3728c142deb718c859b33f134ba58db"
server_key_version: "encryptionkey@0"
I20260929 03:27:17.475677 10023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:17.500005 10023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:17.502126 10023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:17.502305 10023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:17.502526 10023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:17.502810 10023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:17.502861 10023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.502900 10023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:17.502928 10023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:17.508708 10023 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:42419
I20260929 03:27:17.508754 10151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:42419 every 8 connection(s)
I20260929 03:27:17.509030 10023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:17.516129 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10023
I20260929 03:27:17.516228 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:17.516441 6610 external_mini_cluster.cc:1428] Setting key 8d3d2e8f0f1de8c1c0425216e8f858d0
I20260929 03:27:17.517980 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.509617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40183 (local address 127.6.116.190:45177)
0929 03:27:17.509687 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:17.509690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:17.509856 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:27:17.510339 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:17.510340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:17.510351 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:17.510407 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:17.511349 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.511860 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.512824 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.513039 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.513122 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.514992 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:17.514993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:17.514993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:17.515000 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:17.516485 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.516982 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.516983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.516983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.517051 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.517341 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.517343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.517343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.517464 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:17.517541 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:17.517814 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 03:27:17.517894 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:27:17.518163 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.509598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45177 (local address 127.6.116.131:40183)
0929 03:27:17.509740 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:17.509786 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:17.510238 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:17.510469 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:17.510473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:17.510998 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:17.511243 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.511247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.511952 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.511954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.512718 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:17.512719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.513129 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.513130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:17.513244 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.513619 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:17.513637 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:17.514887 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:17.516581 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.516586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.516588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.516904 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.517119 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:17.517120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:17.517121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:17.517256 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:17.517532 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:17.517538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:17.517678 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 03:27:17.517777 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 03:27:17.517922 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260929 03:27:17.518584 10152 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45177
I20260929 03:27:17.518656 10152 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:17.518780 10152 heartbeater.cc:511] Master 127.6.116.190:45177 requested a full tablet report, sending...
I20260929 03:27:17.519126 9693 ts_manager.cc:194] Registered new tserver with Master: 032687e7e3f445a48f43cfeed1458d79 (127.6.116.131:42419)
I20260929 03:27:17.519513 9693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:40183
I20260929 03:27:17.528141 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[9648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652437, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:17.549304 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:17.543004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.6.116.190:45177)
0929 03:27:17.543092 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:17.543094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:17.543182 (+ 88us) server_negotiation.cc:407] Connection header received
0929 03:27:17.543221 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:17.543221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:17.543230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:17.543285 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:17.543779 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.544244 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.544896 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:17.545066 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:17.545112 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:17.547152 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:17.547153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:17.547154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:17.547160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:17.548473 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.548756 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.548757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.548757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.548802 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:17.548940 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:17.548940 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:17.548940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:17.549047 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:17.549113 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:17.549193 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:27:17.549246 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 03:27:17.549528 9693 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38860
I20260929 03:27:17.549924 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9755
I20260929 03:27:17.554695 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9889
I20260929 03:27:17.558743 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10023
I20260929 03:27:17.562773 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 9661
2026-09-29T03:27:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1782 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[10164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[10164](info): set up 2 sockets
Sep 29 03:27:17 dist-test-slave-z2zg krb5kdc[10164](info): commencing operation
krb5kdc: starting...
W20260929 03:27:18.576903 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:18 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:18.689839 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38117
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:18.768528 10177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:18.768769 10177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:18.768806 10177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:18.769922 10177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:18.769989 10177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:18.770015 10177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:18.770046 10177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:18.770071 10177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:18.771600 10177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:18.771869 10177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:18.772087 10177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:18.776839 10182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:18.776856 10183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:18.776844 10185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:18.777107 10177 server_base.cc:1034] running on GCE node
I20260929 03:27:18.777383 10177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:18.777679 10177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:18.778847 10177 hybrid_clock.cc:648] HybridClock initialized: now 1790652438778834 us; error 41 us; skew 500 ppm
Sep 29 03:27:18 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:18.786626 10177 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:18.787159 10177 webserver.cc:460] Webserver started at http://127.6.116.190:38773/ using document root <none> and password file <none>
I20260929 03:27:18.787357 10177 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:18.787398 10177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:18.787485 10177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:18.788483 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "5e998fb62cb44b4893c6cd9703a93265"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "172a9f7589b31b034f95313dcbf61633"
server_key_iv: "8a213543ae9f1548b057ebca3b72b58c"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.788805 10177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "5e998fb62cb44b4893c6cd9703a93265"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "172a9f7589b31b034f95313dcbf61633"
server_key_iv: "8a213543ae9f1548b057ebca3b72b58c"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.790021 10177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:18.790707 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:18.790864 10177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:18.790946 10177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "5e998fb62cb44b4893c6cd9703a93265"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "172a9f7589b31b034f95313dcbf61633"
server_key_iv: "8a213543ae9f1548b057ebca3b72b58c"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.791002 10177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:18.807868 10177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:18.810063 10177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:18.810185 10177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:18.813958 10177 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38117
I20260929 03:27:18.814013 10244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38117 every 8 connection(s)
I20260929 03:27:18.814484 10177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:18.815119 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:18.817358 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265: Bootstrap starting.
I20260929 03:27:18.817940 10245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:18.818243 10245 log.cc:824] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:18.818914 10245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265: No bootstrap required, opened a new log
I20260929 03:27:18.820497 10245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } }
I20260929 03:27:18.820659 10245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:18.820709 10245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e998fb62cb44b4893c6cd9703a93265, State: Initialized, Role: FOLLOWER
I20260929 03:27:18.820890 10245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } }
I20260929 03:27:18.820989 10245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:18.821033 10245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:18.821082 10245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:18.821928 10245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } }
I20260929 03:27:18.822043 10245 leader_election.cc:302] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e998fb62cb44b4893c6cd9703a93265; no voters:
I20260929 03:27:18.822232 10245 leader_election.cc:288] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:18.822281 10248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:18.822476 10245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:18.822497 10248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [term 1 LEADER]: Becoming Leader. State: Replica: 5e998fb62cb44b4893c6cd9703a93265, State: Running, Role: LEADER
I20260929 03:27:18.822579 10248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } }
I20260929 03:27:18.823029 10249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e998fb62cb44b4893c6cd9703a93265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } } }
I20260929 03:27:18.823154 10249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:18.823333 10250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e998fb62cb44b4893c6cd9703a93265. Latest consensus state: current_term: 1 leader_uuid: "5e998fb62cb44b4893c6cd9703a93265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e998fb62cb44b4893c6cd9703a93265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38117 } } }
I20260929 03:27:18.823395 10257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:18.823424 10250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:18.823917 10257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:18.824325 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10177
I20260929 03:27:18.824429 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:18.824600 6610 external_mini_cluster.cc:1428] Setting key 3d00b55fa399312965bf1b17e1dc3c19
I20260929 03:27:18.825645 10257 catalog_manager.cc:1347] Generated new cluster ID: 7f5e24e0cec6451684b38dbb5c703b05
I20260929 03:27:18.825698 10257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:18 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:18.834153 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:18.825026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.6.116.190:38117)
0929 03:27:18.825274 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:18.825283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:18.825319 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:27:18.825424 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:18.825429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:18.825471 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:18.826287 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:18.826358 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:18.827379 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:18.828117 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:18.828544 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:18.828676 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:18.830869 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:18.830873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:18.830873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:18.830887 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:18.832541 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:18.832922 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:18.832925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:18.832926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:18.833030 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:18.833237 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:18.833238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:18.833238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:18.833477 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:18.833639 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:18.833752 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:27:18.833984 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260929 03:27:18.849023 10257 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:18.849541 10257 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:18.860961 10257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e998fb62cb44b4893c6cd9703a93265: Generated new TSK 0
I20260929 03:27:18.861158 10257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:18.887630 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:18.968217 10271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:18.968469 10271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:18.968508 10271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:18.969651 10271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:18.969718 10271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:18.969762 10271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:18.971262 10271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:18.971513 10271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:18.971728 10271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:18.976480 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:18.976486 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:18.976486 10279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:18.976725 10271 server_base.cc:1034] running on GCE node
I20260929 03:27:18.976943 10271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:18.977175 10271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:18.978338 10271 hybrid_clock.cc:648] HybridClock initialized: now 1790652438978332 us; error 36 us; skew 500 ppm
Sep 29 03:27:18 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:18.981210 10271 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:18.981609 10271 webserver.cc:460] Webserver started at http://127.6.116.129:43211/ using document root <none> and password file <none>
I20260929 03:27:18.981832 10271 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:18.981897 10271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:18.982064 10271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:18.983006 10271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "c22a8c00a3264c5faeb8cce6be56f8ec"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "378512076cf3ed6c122a53f5ad35cced"
server_key_iv: "9f259ced82e9213cfd68c9f5bd8204ca"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.983415 10271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "c22a8c00a3264c5faeb8cce6be56f8ec"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "378512076cf3ed6c122a53f5ad35cced"
server_key_iv: "9f259ced82e9213cfd68c9f5bd8204ca"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.984700 10271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:18.985358 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:18.985525 10271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:18.985620 10271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "c22a8c00a3264c5faeb8cce6be56f8ec"
format_stamp: "Formatted at 2026-09-29 03:27:18 on dist-test-slave-z2zg"
server_key: "378512076cf3ed6c122a53f5ad35cced"
server_key_iv: "9f259ced82e9213cfd68c9f5bd8204ca"
server_key_version: "encryptionkey@0"
I20260929 03:27:18.985706 10271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:18.994560 10271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:18.996323 10271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:18.996471 10271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:18.996680 10271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:18.996956 10271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:18.997009 10271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:18.997049 10271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:18.997076 10271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.002467 10271 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:35155
I20260929 03:27:19.002563 10399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:35155 every 8 connection(s)
I20260929 03:27:19.002833 10271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:19.012006 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.003559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:45351 (local address 127.6.116.190:38117)
0929 03:27:19.003599 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:19.003602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:19.003778 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:27:19.004735 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:19.004736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:19.004751 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:19.004822 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:19.005431 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.005934 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.006952 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.007278 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.007354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.009396 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:19.009398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:19.009398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:19.009406 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:19.010661 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.011071 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.011072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.011072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.011136 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.011373 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.011374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.011375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.011488 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:19.011569 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:19.011831 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 03:27:19.011917 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:19.012176 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.003534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38117 (local address 127.6.116.129:45351)
0929 03:27:19.003656 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:19.003699 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:19.004589 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:19.004903 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:19.004907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:19.005157 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:19.005336 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.005340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.006032 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.006033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.006859 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.006861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.007373 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.007374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.007493 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.007988 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:19.007996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:19.009299 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:19.010744 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.010750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.010752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.010961 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.011202 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.011204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.011204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.011305 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.011561 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:19.011567 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:19.011695 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 03:27:19.011802 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 03:27:19.011944 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260929 03:27:19.012516 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10271
I20260929 03:27:19.012619 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:19.012616 10400 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38117
I20260929 03:27:19.012718 10400 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:19.012843 6610 external_mini_cluster.cc:1428] Setting key 1daf382d46d9c746380079df871fe6c7
I20260929 03:27:19.012884 10400 heartbeater.cc:511] Master 127.6.116.190:38117 requested a full tablet report, sending...
I20260929 03:27:19.013477 10208 ts_manager.cc:194] Registered new tserver with Master: c22a8c00a3264c5faeb8cce6be56f8ec (127.6.116.129:35155)
I20260929 03:27:19.014359 10208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:45351
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:19.037750 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:19.117048 10405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:19.117242 10405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:19.117265 10405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:19.118323 10405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:19.118358 10405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:19.118383 10405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:19.119875 10405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:19.120054 10405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:19.120213 10405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:19.125089 10411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:19.125105 10410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:19.125106 10413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:19.125418 10405 server_base.cc:1034] running on GCE node
I20260929 03:27:19.125609 10405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:19.125813 10405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:19.126931 10405 hybrid_clock.cc:648] HybridClock initialized: now 1790652439126918 us; error 27 us; skew 500 ppm
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:19.129350 10405 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:19.129810 10405 webserver.cc:460] Webserver started at http://127.6.116.130:35413/ using document root <none> and password file <none>
I20260929 03:27:19.130026 10405 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:19.130093 10405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:19.130244 10405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:19.131296 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "d347751085544f0e8f13780b25529993"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "376591e209b751db895023ef9c753c54"
server_key_iv: "7a02d4369d5ccf8c9deced7a2922c93e"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.131640 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "d347751085544f0e8f13780b25529993"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "376591e209b751db895023ef9c753c54"
server_key_iv: "7a02d4369d5ccf8c9deced7a2922c93e"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.132881 10405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:19.133584 10420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.133766 10405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:19.133864 10405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "d347751085544f0e8f13780b25529993"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "376591e209b751db895023ef9c753c54"
server_key_iv: "7a02d4369d5ccf8c9deced7a2922c93e"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.133940 10405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:19.147974 10405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:19.150161 10405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:19.150319 10405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:19.150539 10405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:19.150856 10405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:19.150918 10405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.150949 10405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:19.150977 10405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.156525 10405 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:37907
I20260929 03:27:19.156562 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:37907 every 8 connection(s)
I20260929 03:27:19.156864 10405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:19.162042 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10405
I20260929 03:27:19.162150 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:19.162326 6610 external_mini_cluster.cc:1428] Setting key 1d4fbbc8239d7bf1a37a09c5b65f167e
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:19.166333 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.157552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:58597 (local address 127.6.116.190:38117)
0929 03:27:19.157596 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:19.157598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:19.157789 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:27:19.158633 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:19.158634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:19.158644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:19.158701 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:19.159343 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.159828 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.160741 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.160926 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.160977 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.163108 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:19.163110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:19.163110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:19.163118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:19.164946 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.165357 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.165358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.165359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.165425 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.165688 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.165689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.165690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.165809 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:19.165882 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:19.166141 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:27:19.166226 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:19.166531 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.157496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38117 (local address 127.6.116.130:58597)
0929 03:27:19.157657 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:19.157696 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:19.158543 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:19.158779 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:19.158783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:19.159012 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:19.159264 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.159269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.159905 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.159905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.160661 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.160663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.160995 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.160995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.161094 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.161519 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:19.161526 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:19.163000 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:19.165040 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.165045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.165047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.165263 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.165493 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.165493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.165494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.165605 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.165883 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:19.165890 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:19.166015 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:27:19.166121 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:27:19.166292 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260929 03:27:19.167084 10534 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38117
I20260929 03:27:19.167179 10534 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:19.167373 10534 heartbeater.cc:511] Master 127.6.116.190:38117 requested a full tablet report, sending...
I20260929 03:27:19.167747 10208 ts_manager.cc:194] Registered new tserver with Master: d347751085544f0e8f13780b25529993 (127.6.116.130:37907)
I20260929 03:27:19.168115 10208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:58597
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:19.185670 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:19.264184 10539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:19.264370 10539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:19.264451 10539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:19.265575 10539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:19.265617 10539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:19.265647 10539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:19.267131 10539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:19.267362 10539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:19.267586 10539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:19.272328 10544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:19.272379 10545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:19.272523 10539 server_base.cc:1034] running on GCE node
W20260929 03:27:19.272332 10547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:19.272774 10539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:19.273010 10539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:19.274148 10539 hybrid_clock.cc:648] HybridClock initialized: now 1790652439274136 us; error 29 us; skew 500 ppm
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:19.276540 10539 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:19.276963 10539 webserver.cc:460] Webserver started at http://127.6.116.131:43763/ using document root <none> and password file <none>
I20260929 03:27:19.277155 10539 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:19.277195 10539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:19.277278 10539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:19.278203 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "e96d7930f82d4ce9a2c6375fa9ce4803"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "f6041c23a58f7d40c3ada189432412fe"
server_key_iv: "fe7f2d4e6fd7c71aeee92116804fba05"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.278481 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "e96d7930f82d4ce9a2c6375fa9ce4803"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "f6041c23a58f7d40c3ada189432412fe"
server_key_iv: "fe7f2d4e6fd7c71aeee92116804fba05"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.279783 10539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:19.280442 10554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.280619 10539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:19.280725 10539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "e96d7930f82d4ce9a2c6375fa9ce4803"
format_stamp: "Formatted at 2026-09-29 03:27:19 on dist-test-slave-z2zg"
server_key: "f6041c23a58f7d40c3ada189432412fe"
server_key_iv: "fe7f2d4e6fd7c71aeee92116804fba05"
server_key_version: "encryptionkey@0"
I20260929 03:27:19.280802 10539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:19.288165 10539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:19.289958 10539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:19.290097 10539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:19.290308 10539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:19.290589 10539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:19.290659 10539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.290699 10539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:19.290724 10539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:19.296196 10539 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:43243
I20260929 03:27:19.296270 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:43243 every 8 connection(s)
I20260929 03:27:19.296566 10539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:19.299847 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10539
I20260929 03:27:19.299962 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:19.300158 6610 external_mini_cluster.cc:1428] Setting key dc2e36098fa5576ae9878ba3690e38d4
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:19.305194 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.297217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:56837 (local address 127.6.116.190:38117)
0929 03:27:19.297279 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:19.297281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:19.297442 (+ 161us) server_negotiation.cc:407] Connection header received
0929 03:27:19.298267 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:19.298268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:19.298277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:19.298330 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:19.298908 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.299447 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.300523 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.300740 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.300826 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.302654 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:19.302655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:19.302656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:19.302663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:19.303877 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.304287 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.304288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.304289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.304346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.304586 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.304587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.304588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.304703 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:19.304774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:19.305032 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:27:19.305105 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:27:19.305379 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.297187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38117 (local address 127.6.116.131:56837)
0929 03:27:19.297331 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:19.297369 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:19.298184 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:19.298377 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:19.298379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:19.298648 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:19.298828 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.298832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.299521 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.299522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.300436 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:19.300438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.300832 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.300833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:19.300949 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.301326 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:19.301333 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:19.302553 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:19.303970 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.303976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.303978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.304194 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.304411 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:19.304412 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:19.304413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:19.304513 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:19.304779 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:19.304785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:19.304906 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 03:27:19.305009 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 03:27:19.305164 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260929 03:27:19.305852 10668 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38117
I20260929 03:27:19.305944 10668 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:19.306115 10668 heartbeater.cc:511] Master 127.6.116.190:38117 requested a full tablet report, sending...
I20260929 03:27:19.306592 10208 ts_manager.cc:194] Registered new tserver with Master: e96d7930f82d4ce9a2c6375fa9ce4803 (127.6.116.131:43243)
I20260929 03:27:19.306939 10208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:56837
I20260929 03:27:19.311646 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:19.333150 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.325987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.6.116.129:35155)
0929 03:27:19.326162 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:19.326166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:19.326176 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:19.326198 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:19.326200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:19.326212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:19.326287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:19.326799 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.327574 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.328256 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.328538 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.328588 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.330742 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:19.330748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:19.330749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:19.330755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:19.332166 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.332462 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.332464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.332465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.332524 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.332692 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.332692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.332693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.332806 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:19.332900 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:19.332964 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:27:19.333021 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:19.341007 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:19.334388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.6.116.190:38117)
0929 03:27:19.334444 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:19.334446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:19.334499 (+ 53us) server_negotiation.cc:407] Connection header received
0929 03:27:19.334551 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:19.334552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:19.334559 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:19.334601 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:19.335016 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.335510 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.336139 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:19.336307 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:19.336354 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:19.338500 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:19.338502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:19.338502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:19.338510 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:19.340025 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.340350 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.340350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.340351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.340398 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:19.340583 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:19.340583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:19.340584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:19.340712 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:19.340781 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:19.340869 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:27:19.340922 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 03:27:19.341254 10208 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58678
I20260929 03:27:19.341607 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10271
I20260929 03:27:19.346487 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10405
I20260929 03:27:19.350538 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10539
I20260929 03:27:19.354638 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10177
2026-09-29T03:27:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1791 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10680](info): set up 2 sockets
Sep 29 03:27:19 dist-test-slave-z2zg krb5kdc[10680](info): commencing operation
krb5kdc: starting...
W20260929 03:27:20.384902 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:20.498462 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34325
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:34325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:20.575969 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:20.576176 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:20.576213 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:20.577275 10693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:20.577337 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:20.577364 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:20.577394 10693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:20.577419 10693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:20.578814 10693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:34325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:20.579082 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:20.579293 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:20.583892 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:20.584033 10693 server_base.cc:1034] running on GCE node
W20260929 03:27:20.583884 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:20.583880 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:20.584333 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:20.584622 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:20.585773 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790652440585762 us; error 27 us; skew 500 ppm
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:20.588191 10693 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:20.588498 10693 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:20.588551 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:20.588711 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:20.589708 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "50adb03536af49798342feaaed1d3b47"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "7ae415566aef02bd57463c8efdaf20b5"
server_key_iv: "01af7d1001b2e562c5e0901212e35148"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.590058 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "50adb03536af49798342feaaed1d3b47"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "7ae415566aef02bd57463c8efdaf20b5"
server_key_iv: "01af7d1001b2e562c5e0901212e35148"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.591286 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:20.591930 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.592108 10693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:20.592175 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "50adb03536af49798342feaaed1d3b47"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "7ae415566aef02bd57463c8efdaf20b5"
server_key_iv: "01af7d1001b2e562c5e0901212e35148"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.592257 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:20.618539 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:20.620391 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:20.620540 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:20.624382 10693 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:34325
I20260929 03:27:20.624408 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:34325 every 8 connection(s)
I20260929 03:27:20.624681 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:20.625536 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:20.627687 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: Bootstrap starting.
I20260929 03:27:20.628194 10760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:20.628417 10760 log.cc:824] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:20.628999 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: No bootstrap required, opened a new log
I20260929 03:27:20.630434 10760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } }
I20260929 03:27:20.630569 10760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:20.630591 10760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50adb03536af49798342feaaed1d3b47, State: Initialized, Role: FOLLOWER
I20260929 03:27:20.630723 10760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } }
I20260929 03:27:20.630805 10760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:20.630826 10760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:20.630854 10760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:20.631608 10760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } }
I20260929 03:27:20.631688 10760 leader_election.cc:302] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50adb03536af49798342feaaed1d3b47; no voters:
I20260929 03:27:20.631801 10760 leader_election.cc:288] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:20.631880 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:20.632058 10760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:20.632072 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [term 1 LEADER]: Becoming Leader. State: Replica: 50adb03536af49798342feaaed1d3b47, State: Running, Role: LEADER
I20260929 03:27:20.632261 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } }
I20260929 03:27:20.632607 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50adb03536af49798342feaaed1d3b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } } }
I20260929 03:27:20.632602 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50adb03536af49798342feaaed1d3b47. Latest consensus state: current_term: 1 leader_uuid: "50adb03536af49798342feaaed1d3b47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50adb03536af49798342feaaed1d3b47" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34325 } } }
I20260929 03:27:20.632702 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:20.632709 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:20.633047 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10693
I20260929 03:27:20.633173 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:20.633179 10772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:20.633363 6610 external_mini_cluster.cc:1428] Setting key 50ce3f7c40c528977d6c16a4d7850a9f
I20260929 03:27:20.633896 10772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:20.635653 10772 catalog_manager.cc:1347] Generated new cluster ID: c29bee4f1e48459a9dc10c532f5570c3
I20260929 03:27:20.635704 10772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:20.640481 10772 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:20.641139 10772 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:20.643952 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:20.633760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.6.116.190:34325)
0929 03:27:20.633937 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:20.633947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:20.635126 (+ 1179us) server_negotiation.cc:407] Connection header received
0929 03:27:20.635333 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:20.635347 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:20.635389 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:20.635528 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:20.636209 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.637118 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.637852 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.638273 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.638387 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:20.640823 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:20.640826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:20.640827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:20.640837 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:20.642488 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.642864 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.642867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.642868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.642953 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.643163 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.643164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.643164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.643366 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:20.643545 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:20.643625 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 03:27:20.643775 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260929 03:27:20.646836 10772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: Generated new TSK 0
I20260929 03:27:20.647011 10772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:20.675282 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:20.791451 10786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:20.791719 10786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:20.791759 10786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:20.793418 10786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:20.793480 10786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:20.793520 10786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:20.795675 10786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:20.795900 10786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:20.796152 10786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:20.801210 10791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:20.801218 10792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:20.801188 10794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:20.801627 10786 server_base.cc:1034] running on GCE node
I20260929 03:27:20.801819 10786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:20.802063 10786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:20.803215 10786 hybrid_clock.cc:648] HybridClock initialized: now 1790652440803197 us; error 30 us; skew 500 ppm
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:20.806178 10786 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:20.806555 10786 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:20.806631 10786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:20.806782 10786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:20.807883 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "7ff1fe3ec65e48439113955fcdb05254"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "202d31bff725162e97ac523bef15b21e"
server_key_iv: "b83ea4e265d4bee88b23f535ff66bce5"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.808271 10786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "7ff1fe3ec65e48439113955fcdb05254"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "202d31bff725162e97ac523bef15b21e"
server_key_iv: "b83ea4e265d4bee88b23f535ff66bce5"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.809614 10786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:20.810350 10800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.810515 10786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:20.810616 10786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "7ff1fe3ec65e48439113955fcdb05254"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "202d31bff725162e97ac523bef15b21e"
server_key_iv: "b83ea4e265d4bee88b23f535ff66bce5"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.810698 10786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:20.819885 10786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:20.822099 10786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:20.822256 10786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:20.822467 10786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:20.822750 10786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:20.822803 10786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.822840 10786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:20.822866 10786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.828853 10786 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:39145
I20260929 03:27:20.828929 10913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:39145 every 8 connection(s)
I20260929 03:27:20.829173 10786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:20.830212 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10786
I20260929 03:27:20.830295 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:20.830489 6610 external_mini_cluster.cc:1428] Setting key 0a071b95dd0f3c04bd867811c53f9834
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:20.839028 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:20.829843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:55787 (local address 127.6.116.190:34325)
0929 03:27:20.829887 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:20.829889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:20.830068 (+ 179us) server_negotiation.cc:407] Connection header received
0929 03:27:20.830658 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:20.830659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:20.830683 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:20.830734 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:20.831758 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.832308 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.833474 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.833802 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.833873 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:20.836088 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:20.836091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:20.836091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:20.836100 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:20.837720 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.838103 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.838105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.838105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.838168 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.838393 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.838394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.838394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.838529 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:20.838604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:20.838834 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 03:27:20.838923 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:20.839037 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:20.829806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34325 (local address 127.6.116.129:55787)
0929 03:27:20.829938 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:20.829985 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:20.830557 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:20.830798 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:20.830802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:20.831195 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:20.831654 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:20.831658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.832442 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.832444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:20.833402 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:20.833404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.833884 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.833885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:20.833986 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:20.834468 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:20.834476 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:20.835990 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:20.837813 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:20.837819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:20.837821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:20.838029 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:20.838237 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:20.838237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:20.838238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:20.838327 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:20.838597 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:20.838601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:20.838706 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:27:20.838782 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:20.838890 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260929 03:27:20.839547 10914 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34325
I20260929 03:27:20.839641 10914 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:20.839807 10914 heartbeater.cc:511] Master 127.6.116.190:34325 requested a full tablet report, sending...
I20260929 03:27:20.840232 10724 ts_manager.cc:194] Registered new tserver with Master: 7ff1fe3ec65e48439113955fcdb05254 (127.6.116.129:39145)
I20260929 03:27:20.840939 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:55787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:20.855530 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:20.933233 10919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:20.933404 10919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:20.933426 10919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:20.934556 10919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:20.934593 10919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:20.934621 10919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:20.936013 10919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:20.936190 10919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:20.936342 10919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:20.940900 10925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:20.940902 10927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:20.941099 10919 server_base.cc:1034] running on GCE node
W20260929 03:27:20.940902 10924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:20.941409 10919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:20.941648 10919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:20.942814 10919 hybrid_clock.cc:648] HybridClock initialized: now 1790652440942800 us; error 25 us; skew 500 ppm
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:20.945206 10919 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:20.945528 10919 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:20.945596 10919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:20.945756 10919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:20.946720 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "bd860d803ed1446abd6a425d29669ec2"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "83cc9d40fc71871e7b540cc0dad7370b"
server_key_iv: "1238857cc95f622c06d0d6b57b4ca75f"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.947145 10919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "bd860d803ed1446abd6a425d29669ec2"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "83cc9d40fc71871e7b540cc0dad7370b"
server_key_iv: "1238857cc95f622c06d0d6b57b4ca75f"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.948333 10919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:20.949018 10933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.949177 10919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:20.949272 10919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "bd860d803ed1446abd6a425d29669ec2"
format_stamp: "Formatted at 2026-09-29 03:27:20 on dist-test-slave-z2zg"
server_key: "83cc9d40fc71871e7b540cc0dad7370b"
server_key_iv: "1238857cc95f622c06d0d6b57b4ca75f"
server_key_version: "encryptionkey@0"
I20260929 03:27:20.949355 10919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:20.962347 10919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:20.963995 10919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:20.964157 10919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:20.964366 10919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:20.964651 10919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:20.964705 10919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.964744 10919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:20.964772 10919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:20.970353 10919 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:43461
I20260929 03:27:20.970430 11046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:43461 every 8 connection(s)
I20260929 03:27:20.970680 10919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:20 dist-test-slave-z2zg krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652440, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:20.979292 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:20.971367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:36369 (local address 127.6.116.190:34325)
0929 03:27:20.971435 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:20.971437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:20.971635 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:27:20.972076 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:20.972077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:20.972090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:20.972140 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:20.972968 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.973470 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.974429 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.974654 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.974738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:20.976772 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:20.976774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:20.976774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:20.976782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:20.977947 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.978346 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.978347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.978348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.978405 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:20.978662 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:20.978663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:20.978663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:20.978776 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:20.978847 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:20.979126 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 03:27:20.979203 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:27:20.979497 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:20.971357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34325 (local address 127.6.116.130:36369)
0929 03:27:20.971517 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:20.971546 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:20.971974 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:20.972210 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:20.972213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:20.972438 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:20.972868 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:20.972872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.973559 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.973560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:20.974332 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:20.974334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:20.974748 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:20.974749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:20.974864 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:20.975363 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:20.975371 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:20.976668 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:20.978042 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:20.978049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:20.978051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:20.978273 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:20.978469 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:20.978470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:20.978471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:20.978572 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:20.978849 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:20.978856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:20.978989 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 03:27:20.979107 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:27:20.979256 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260929 03:27:20.979925 11047 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34325
I20260929 03:27:20.980042 11047 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:20.980175 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 10919
I20260929 03:27:20.980194 11047 heartbeater.cc:511] Master 127.6.116.190:34325 requested a full tablet report, sending...
I20260929 03:27:20.980281 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:20.980479 6610 external_mini_cluster.cc:1428] Setting key a9e6b76ad65bad34517e26eaf0fd1d21
I20260929 03:27:20.980634 10724 ts_manager.cc:194] Registered new tserver with Master: bd860d803ed1446abd6a425d29669ec2 (127.6.116.130:43461)
I20260929 03:27:20.981269 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:36369
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:21.003384 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:21.080471 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:21.080655 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:21.080677 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:21.081765 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:21.081804 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:21.081828 11052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:21.083256 11052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:21.083477 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:21.083678 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:21.088248 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.088239 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.088425 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.088406 11052 server_base.cc:1034] running on GCE node
I20260929 03:27:21.088639 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:21.088850 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:21.089975 11052 hybrid_clock.cc:648] HybridClock initialized: now 1790652441089966 us; error 25 us; skew 500 ppm
Sep 29 03:27:21 dist-test-slave-z2zg krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652441, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:21.092276 11052 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:21.092566 11052 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:21.092607 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:21.092770 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:21.093698 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "d2eb48f2e8944947ad2d4cfacd0a786f"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "231fefdffa78d143cdf8eaa457e4d5f5"
server_key_iv: "bc18edcd0b4e4d64f57d745874409316"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.094022 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "d2eb48f2e8944947ad2d4cfacd0a786f"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "231fefdffa78d143cdf8eaa457e4d5f5"
server_key_iv: "bc18edcd0b4e4d64f57d745874409316"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.095455 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:21.096210 11066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.096390 11052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:21.096469 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "d2eb48f2e8944947ad2d4cfacd0a786f"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "231fefdffa78d143cdf8eaa457e4d5f5"
server_key_iv: "bc18edcd0b4e4d64f57d745874409316"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.096532 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:21.109282 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:21.110903 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:21.111035 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:21.111312 11052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:21.111578 11052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:21.111610 11052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.111635 11052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:21.111649 11052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.117234 11052 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:45217
I20260929 03:27:21.117312 11179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:45217 every 8 connection(s)
I20260929 03:27:21.117542 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:21 dist-test-slave-z2zg krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652441, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:21.125680 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.118187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:49151 (local address 127.6.116.190:34325)
0929 03:27:21.118258 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.118261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.118488 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:27:21.118884 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.118885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.118894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.118945 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.119806 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.120275 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.121158 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.121329 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.121412 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.123256 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.123258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.123258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:21.123264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.124626 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:21.124952 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:21.124953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:21.124953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:21.124999 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:21.125191 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:21.125192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:21.125192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:21.125288 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.125334 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.125529 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.125587 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:27:21.125804 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.118183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34325 (local address 127.6.116.131:49151)
0929 03:27:21.118336 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.118378 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.118810 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.118993 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.118996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.119299 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.119734 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.119738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.120346 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.120347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.121091 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.121093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.121422 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.121423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.121509 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.121878 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:21.121885 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.123181 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.124695 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:21.124700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:21.124701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:21.124891 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:21.125054 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:21.125054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:21.125055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:21.125140 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:21.125340 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.125344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.125440 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.125518 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.125631 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260929 03:27:21.126247 11180 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34325
I20260929 03:27:21.126339 11180 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:21.126466 11180 heartbeater.cc:511] Master 127.6.116.190:34325 requested a full tablet report, sending...
I20260929 03:27:21.126803 10724 ts_manager.cc:194] Registered new tserver with Master: d2eb48f2e8944947ad2d4cfacd0a786f (127.6.116.131:45217)
I20260929 03:27:21.127179 10724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:49151
I20260929 03:27:21.127374 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11052
I20260929 03:27:21.127475 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:21.127666 6610 external_mini_cluster.cc:1428] Setting key 0935c5f5d052fb69e7d2c08e7dceffdf
I20260929 03:27:21.128496 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:27:21.135452 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.129686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.6.116.190:34325)
0929 03:27:21.129719 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.129722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.129858 (+ 136us) server_negotiation.cc:407] Connection header received
0929 03:27:21.129879 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.129880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.129886 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.129933 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.130408 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.130824 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.131549 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.131730 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.131794 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.133161 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.133162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.133163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:21.133169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.134548 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:21.134821 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:21.134821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:21.134821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:21.134867 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:21.135001 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:21.135001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:21.135001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:21.135166 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.135237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.135320 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.135372 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:27:21.136704 10724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:21.138038 10724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:21.144057 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.6.116.130:43461)
0929 03:27:21.140412 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.140415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.140739 (+ 324us) server_negotiation.cc:407] Connection header received
0929 03:27:21.140805 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.140807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.140821 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.140889 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:21.141045 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.141683 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143180 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143638 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143674 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.143705 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.143931 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.143982 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":102,"threads_started":1}
I20260929 03:27:21.144106 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45217 (local address 127.0.0.1:53356)
0929 03:27:21.140090 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.140123 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.140196 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.140472 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.140475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.140478 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:21.140532 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.140535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.141891 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.141892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.143086 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.143089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143863 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.143937 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.143940 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.143979 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.144032 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":52,"thread_start_us":43,"threads_started":1}
I20260929 03:27:21.144260 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.6.116.129:39145)
0929 03:27:21.140537 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.140541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.140606 (+ 65us) server_negotiation.cc:407] Connection header received
0929 03:27:21.140693 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.140695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.140710 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.140780 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:21.140974 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.141996 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143081 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143778 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143848 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.143900 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.144148 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.144188 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260929 03:27:21.144323 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39145 (local address 127.0.0.1:46000)
0929 03:27:21.140554 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.140563 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.140618 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.140870 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.140871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.140873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:21.140912 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.140914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.142098 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.142099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.142976 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.142978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143937 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.144009 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.144023 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.144076 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.144110 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":50,"threads_started":1}
I20260929 03:27:21.144448 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.6.116.131:45217)
0929 03:27:21.140287 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.140292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.140302 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:21.140334 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.140336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.140350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.140462 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:21.140594 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.141803 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143170 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143784 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143847 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.144035 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.144059 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.144096 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
I20260929 03:27:21.144438 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.140051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:43461 (local address 127.0.0.1:33286)
0929 03:27:21.140688 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.140697 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.140742 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.140964 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.140965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.140967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:21.140994 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.140995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.141779 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.141780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.143090 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.143092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.143752 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.143753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.143801 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.143809 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.144300 (+ 491us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.144344 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":34,"thread_start_us":50,"threads_started":1}
I20260929 03:27:21.145064 10848 tablet_service.cc:1467] Processing CreateTablet for tablet bc7e07a90b074ea1810a2af33a35237b (DEFAULT_TABLE table=test-table [id=9f933a0077794c7bb8f92e5bc99ab9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.145109 11114 tablet_service.cc:1467] Processing CreateTablet for tablet bc7e07a90b074ea1810a2af33a35237b (DEFAULT_TABLE table=test-table [id=9f933a0077794c7bb8f92e5bc99ab9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.145236 10981 tablet_service.cc:1467] Processing CreateTablet for tablet bc7e07a90b074ea1810a2af33a35237b (DEFAULT_TABLE table=test-table [id=9f933a0077794c7bb8f92e5bc99ab9de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.145351 11114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7e07a90b074ea1810a2af33a35237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.145387 10848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7e07a90b074ea1810a2af33a35237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.145493 10981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc7e07a90b074ea1810a2af33a35237b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.147992 11194 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Bootstrap starting.
I20260929 03:27:21.148554 11194 tablet_bootstrap.cc:654] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.148581 11196 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Bootstrap starting.
I20260929 03:27:21.149130 11194 log.cc:824] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.149241 11196 tablet_bootstrap.cc:654] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.149813 11196 log.cc:824] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.149905 11194 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: No bootstrap required, opened a new log
I20260929 03:27:21.149973 11194 ts_tablet_manager.cc:1397] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:21.150144 11195 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Bootstrap starting.
I20260929 03:27:21.150458 11196 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: No bootstrap required, opened a new log
I20260929 03:27:21.150532 11196 ts_tablet_manager.cc:1397] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:21.150720 11195 tablet_bootstrap.cc:654] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.151454 11195 log.cc:824] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.152166 11195 tablet_bootstrap.cc:492] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: No bootstrap required, opened a new log
I20260929 03:27:21.152091 11194 raft_consensus.cc:348] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.152227 11195 ts_tablet_manager.cc:1397] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:21.152230 11194 raft_consensus.cc:374] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.152262 11194 raft_consensus.cc:729] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2eb48f2e8944947ad2d4cfacd0a786f, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.152391 11194 consensus_queue.cc:260] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.152485 11196 raft_consensus.cc:348] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.152629 11196 raft_consensus.cc:374] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.152640 11194 ts_tablet_manager.cc:1428] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:21.152669 11196 raft_consensus.cc:729] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff1fe3ec65e48439113955fcdb05254, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.152738 11180 heartbeater.cc:503] Master 127.6.116.190:34325 was elected leader, sending a full tablet report...
I20260929 03:27:21.152786 11196 consensus_queue.cc:260] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.153054 11196 ts_tablet_manager.cc:1428] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:27:21.153093 10914 heartbeater.cc:503] Master 127.6.116.190:34325 was elected leader, sending a full tablet report...
I20260929 03:27:21.154134 11195 raft_consensus.cc:348] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.154237 11195 raft_consensus.cc:374] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.154255 11195 raft_consensus.cc:729] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd860d803ed1446abd6a425d29669ec2, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.154349 11195 consensus_queue.cc:260] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.154577 11195 ts_tablet_manager.cc:1428] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:27:21.154636 11047 heartbeater.cc:503] Master 127.6.116.190:34325 was elected leader, sending a full tablet report...
W20260929 03:27:21.221180 11048 tablet.cc:2369] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:21.329893 10915 tablet.cc:2369] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:21.368160 11181 tablet.cc:2369] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:21.397533 11201 raft_consensus.cc:482] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:21.397639 11201 raft_consensus.cc:504] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.397929 11201 leader_election.cc:288] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2eb48f2e8944947ad2d4cfacd0a786f (127.6.116.131:45217), bd860d803ed1446abd6a425d29669ec2 (127.6.116.130:43461)
I20260929 03:27:21.400919 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.398172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:41279 (local address 127.6.116.131:45217)
0929 03:27:21.398210 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.398213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.398325 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:27:21.398425 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.398426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.398441 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.398501 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:21.398659 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.399223 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400235 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.400593 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400689 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.400697 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.400822 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.400856 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:27:21.400975 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.398088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45217 (local address 127.6.116.129:41279)
0929 03:27:21.398236 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.398249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.398342 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.398514 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.398515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.398516 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:21.398572 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.398575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.399288 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.399289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.400148 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.400150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.400694 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.400752 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.400755 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.400818 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.400856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":34,"thread_start_us":72,"threads_started":1}
I20260929 03:27:21.400969 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.398152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:59099 (local address 127.6.116.130:43461)
0929 03:27:21.398213 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.398215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.398345 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:27:21.398413 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.398414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.398424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.398513 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:21.398698 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.399259 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400208 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.400613 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.400714 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.400870 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.400903 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:21.401041 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.398088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:43461 (local address 127.6.116.129:59099)
0929 03:27:21.398266 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.398276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.398360 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.398524 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.398525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.398527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:21.398604 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.398621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.399325 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.399326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.400100 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:21.400102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.400706 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.400706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:21.400788 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.400792 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.400862 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.400899 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":9,"thread_start_us":63,"threads_started":1}
I20260929 03:27:21.401335 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7e07a90b074ea1810a2af33a35237b" candidate_uuid: "7ff1fe3ec65e48439113955fcdb05254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" is_pre_election: true
I20260929 03:27:21.401399 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7e07a90b074ea1810a2af33a35237b" candidate_uuid: "7ff1fe3ec65e48439113955fcdb05254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd860d803ed1446abd6a425d29669ec2" is_pre_election: true
I20260929 03:27:21.401475 11134 raft_consensus.cc:2454] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff1fe3ec65e48439113955fcdb05254 in term 0.
I20260929 03:27:21.401511 11001 raft_consensus.cc:2454] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff1fe3ec65e48439113955fcdb05254 in term 0.
I20260929 03:27:21.401706 10804 leader_election.cc:302] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff1fe3ec65e48439113955fcdb05254, d2eb48f2e8944947ad2d4cfacd0a786f; no voters:
I20260929 03:27:21.401842 11201 raft_consensus.cc:2781] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:21.401895 11201 raft_consensus.cc:482] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:21.401918 11201 raft_consensus.cc:3037] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:21.402596 11201 raft_consensus.cc:504] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.402729 11201 leader_election.cc:288] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [CANDIDATE]: Term 1 election: Requested vote from peers d2eb48f2e8944947ad2d4cfacd0a786f (127.6.116.131:45217), bd860d803ed1446abd6a425d29669ec2 (127.6.116.130:43461)
I20260929 03:27:21.402930 11134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7e07a90b074ea1810a2af33a35237b" candidate_uuid: "7ff1fe3ec65e48439113955fcdb05254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f"
I20260929 03:27:21.402941 11001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc7e07a90b074ea1810a2af33a35237b" candidate_uuid: "7ff1fe3ec65e48439113955fcdb05254" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd860d803ed1446abd6a425d29669ec2"
I20260929 03:27:21.403019 11134 raft_consensus.cc:3037] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:21.403020 11001 raft_consensus.cc:3037] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:21.403899 11134 raft_consensus.cc:2454] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff1fe3ec65e48439113955fcdb05254 in term 1.
I20260929 03:27:21.403899 11001 raft_consensus.cc:2454] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff1fe3ec65e48439113955fcdb05254 in term 1.
I20260929 03:27:21.404063 10804 leader_election.cc:302] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff1fe3ec65e48439113955fcdb05254, d2eb48f2e8944947ad2d4cfacd0a786f; no voters:
I20260929 03:27:21.404176 11201 raft_consensus.cc:2781] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:21.404316 11201 raft_consensus.cc:686] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff1fe3ec65e48439113955fcdb05254, State: Running, Role: LEADER
I20260929 03:27:21.404425 11201 consensus_queue.cc:237] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } }
I20260929 03:27:21.405241 10724 catalog_manager.cc:5495] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff1fe3ec65e48439113955fcdb05254 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "7ff1fe3ec65e48439113955fcdb05254" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff1fe3ec65e48439113955fcdb05254" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:21.423321 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.421291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.6.116.129:39145)
0929 03:27:21.421340 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.421341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.421349 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:21.421379 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.421380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.421393 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.421451 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:21.421557 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.421983 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.422689 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:21.422857 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:21.422903 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:21.423032 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:21.423170 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:21.423232 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.423264 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.423278 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:21.424849 11134 raft_consensus.cc:1261] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 1 FOLLOWER]: Refusing update from remote peer 7ff1fe3ec65e48439113955fcdb05254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:21.424856 11001 raft_consensus.cc:1261] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 1 FOLLOWER]: Refusing update from remote peer 7ff1fe3ec65e48439113955fcdb05254: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:21.425155 11201 consensus_queue.cc:1035] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2eb48f2e8944947ad2d4cfacd0a786f" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:21.425279 11205 consensus_queue.cc:1035] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd860d803ed1446abd6a425d29669ec2" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:21.427492 11208 mvcc.cc:205] Tried to move back new op lower bound from 7334512400073809920 to 7334512399993188352. Current Snapshot: MvccSnapshot[applied={T|T < 7334512400073809920}]
I20260929 03:27:21.427531 11210 mvcc.cc:205] Tried to move back new op lower bound from 7334512400073809920 to 7334512399993188352. Current Snapshot: MvccSnapshot[applied={T|T < 7334512400073809920}]
I20260929 03:27:21.427701 11209 mvcc.cc:205] Tried to move back new op lower bound from 7334512400073809920 to 7334512399993188352. Current Snapshot: MvccSnapshot[applied={T|T < 7334512400073809920}]
I20260929 03:27:21.429602 10723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:21.429704 10723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:21.430346 10723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50adb03536af49798342feaaed1d3b47: Sending DeleteTablet for 3 replicas of tablet bc7e07a90b074ea1810a2af33a35237b
I20260929 03:27:21.430663 10848 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7e07a90b074ea1810a2af33a35237b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:21 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:46000
I20260929 03:27:21.430732 10981 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7e07a90b074ea1810a2af33a35237b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:21 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:33286
I20260929 03:27:21.430687 11114 tablet_service.cc:1514] Processing DeleteTablet for tablet bc7e07a90b074ea1810a2af33a35237b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:21 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53356
I20260929 03:27:21.430892 11218 tablet_replica.cc:330] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254: stopping tablet replica
I20260929 03:27:21.430977 11218 raft_consensus.cc:2229] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:21.430992 11220 tablet_replica.cc:330] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: stopping tablet replica
I20260929 03:27:21.431095 11220 raft_consensus.cc:2229] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:21.431097 11218 raft_consensus.cc:2258] T bc7e07a90b074ea1810a2af33a35237b P 7ff1fe3ec65e48439113955fcdb05254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:21.431232 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10786
I20260929 03:27:21.431188 11220 raft_consensus.cc:2258] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:21.432567 11220 ts_tablet_manager.cc:1905] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:21.434270 11220 ts_tablet_manager.cc:1918] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:21.434352 11220 log.cc:1196] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/bc7e07a90b074ea1810a2af33a35237b
I20260929 03:27:21.434594 11220 ts_tablet_manager.cc:1939] T bc7e07a90b074ea1810a2af33a35237b P d2eb48f2e8944947ad2d4cfacd0a786f: Deleting consensus metadata
I20260929 03:27:21.434971 10711 catalog_manager.cc:4841] TS d2eb48f2e8944947ad2d4cfacd0a786f (127.6.116.131:45217): tablet bc7e07a90b074ea1810a2af33a35237b (table test-table [id=9f933a0077794c7bb8f92e5bc99ab9de]) successfully deleted
W20260929 03:27:21.436358 10708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:39145 (error 108)
I20260929 03:27:21.436645 11219 tablet_replica.cc:330] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: stopping tablet replica
I20260929 03:27:21.436749 11219 raft_consensus.cc:2229] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:21.436777 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.436657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39145 (local address 127.0.0.1:46008)
0929 03:27:21.436712 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.436746 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 03:27:21.436839 11219 raft_consensus.cc:2258] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 03:27:21.436909 10708 catalog_manager.cc:4584] TS 7ff1fe3ec65e48439113955fcdb05254 (127.6.116.129:39145): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc7e07a90b074ea1810a2af33a35237b: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39145: connect: Connection refused (error 111)
I20260929 03:27:21.436990 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10919
I20260929 03:27:21.437229 11219 ts_tablet_manager.cc:1905] T bc7e07a90b074ea1810a2af33a35237b P bd860d803ed1446abd6a425d29669ec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:21.442157 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11052
I20260929 03:27:21.442312 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.442197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:43461 (local address 127.0.0.1:33292)
0929 03:27:21.442247 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.442278 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:27:21.446411 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 10693
2026-09-29T03:27:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2091 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T03:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:21Z Disabled control of system clock
I20260929 03:27:21.472796 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41981
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:27:21.549685 11225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:21.549862 11225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:21.549901 11225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:21.550918 11225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:21.550951 11225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:21.550962 11225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:21.550976 11225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:21.550987 11225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:21.552253 11225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:21.552423 11225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:21.552655 11225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:21.557185 11233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.557245 11225 server_base.cc:1034] running on GCE node
W20260929 03:27:21.557186 11231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.557264 11230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.557691 11225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:21.557914 11225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:21.559104 11225 hybrid_clock.cc:648] HybridClock initialized: now 1790652441559086 us; error 35 us; skew 500 ppm
I20260929 03:27:21.560361 11225 webserver.cc:460] Webserver started at http://127.6.116.190:40391/ using document root <none> and password file <none>
I20260929 03:27:21.560598 11225 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:21.560675 11225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:21.560828 11225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:21.561770 11225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "049bee76d5bf4bf9bbcef04848881ddf"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "c40dde7c728e7b70b3a862e318491947"
server_key_iv: "50c36c4d22d16f3dcdcb7290f88fe88f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.562096 11225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "049bee76d5bf4bf9bbcef04848881ddf"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "c40dde7c728e7b70b3a862e318491947"
server_key_iv: "50c36c4d22d16f3dcdcb7290f88fe88f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.563350 11225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:21.564019 11239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.564195 11225 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:21.564291 11225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "049bee76d5bf4bf9bbcef04848881ddf"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "c40dde7c728e7b70b3a862e318491947"
server_key_iv: "50c36c4d22d16f3dcdcb7290f88fe88f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.564366 11225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:21.567571 11225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:21.567791 11225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:21.567950 11225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:21.571514 11225 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:41981
I20260929 03:27:21.571563 11291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:41981 every 8 connection(s)
I20260929 03:27:21.571918 11225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:21.572486 11292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.575150 11292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: Bootstrap starting.
I20260929 03:27:21.575711 11292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.576002 11292 log.cc:824] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.576613 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11225
I20260929 03:27:21.576704 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:21.576793 11292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: No bootstrap required, opened a new log
I20260929 03:27:21.576893 6610 external_mini_cluster.cc:1428] Setting key ee27f45658a4515a998248c93263336d
I20260929 03:27:21.578984 11292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } }
I20260929 03:27:21.579200 11292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.579227 11292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049bee76d5bf4bf9bbcef04848881ddf, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.579276 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.577344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54040 (local address 127.6.116.190:41981)
0929 03:27:21.577521 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.577527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.577549 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:27:21.578020 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.578026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.578058 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.578161 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.578780 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.578783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.578784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.578787 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.578821 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.578830 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.578880 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.578968 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.579032 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260929 03:27:21.579365 11292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } }
I20260929 03:27:21.579463 11292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:21.579500 11292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:21.579551 11292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:21.580440 11292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } }
I20260929 03:27:21.580549 11292 leader_election.cc:302] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 049bee76d5bf4bf9bbcef04848881ddf; no voters:
I20260929 03:27:21.580698 11292 leader_election.cc:288] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:21.580753 11297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:21.580894 11297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [term 1 LEADER]: Becoming Leader. State: Replica: 049bee76d5bf4bf9bbcef04848881ddf, State: Running, Role: LEADER
I20260929 03:27:21.580972 11292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:21.580991 11297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } }
I20260929 03:27:21.581288 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 049bee76d5bf4bf9bbcef04848881ddf. Latest consensus state: current_term: 1 leader_uuid: "049bee76d5bf4bf9bbcef04848881ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } } }
I20260929 03:27:21.581298 11299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "049bee76d5bf4bf9bbcef04848881ddf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "049bee76d5bf4bf9bbcef04848881ddf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41981 } } }
I20260929 03:27:21.581372 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:21.581372 11299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:21.581779 11302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:21.582206 11302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:21.583750 11302 catalog_manager.cc:1347] Generated new cluster ID: b4d1cd8142854e93b5c96e91adeedb1a
I20260929 03:27:21.583808 11302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:21.595271 11302 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:21.595410 11302 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:21.606666 11302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: Generated new TSK 0
I20260929 03:27:21.606875 11302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:21.618162 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--builtin_ntp_servers=127.6.116.148:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:27:21.698064 11316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:21.698256 11316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:21.698276 11316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:21.699478 11316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:21.699517 11316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:21.699542 11316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:21.700851 11316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:21.701078 11316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:21.701373 11316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:21.706007 11322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.706019 11324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.706254 11316 server_base.cc:1034] running on GCE node
W20260929 03:27:21.706359 11321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.706575 11316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:21.706848 11316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:21.708015 11316 hybrid_clock.cc:648] HybridClock initialized: now 1790652441708003 us; error 28 us; skew 500 ppm
I20260929 03:27:21.709272 11316 webserver.cc:460] Webserver started at http://127.6.116.129:44677/ using document root <none> and password file <none>
I20260929 03:27:21.709514 11316 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:21.709580 11316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:21.709728 11316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:21.710729 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "df3eff6ecee5463796c83fbe9fdfb6a4"
server_key_iv: "5c0c8af811f21998b1cf3ee2679cfba9"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.711133 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "df3eff6ecee5463796c83fbe9fdfb6a4"
server_key_iv: "5c0c8af811f21998b1cf3ee2679cfba9"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.712395 11316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:21.713095 11330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.713282 11316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:21.713375 11316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "df3eff6ecee5463796c83fbe9fdfb6a4"
server_key_iv: "5c0c8af811f21998b1cf3ee2679cfba9"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.713459 11316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:21.716732 11316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:21.716979 11316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:21.717120 11316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:21.717324 11316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:21.717605 11316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:21.717664 11316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.717702 11316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:21.717730 11316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.723240 11316 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:36671
I20260929 03:27:21.723277 11443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:36671 every 8 connection(s)
I20260929 03:27:21.723574 11316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:21.726606 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.724260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:41117 (local address 127.6.116.190:41981)
0929 03:27:21.724302 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.724304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.724541 (+ 237us) server_negotiation.cc:407] Connection header received
0929 03:27:21.724997 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.724999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.725005 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.725060 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.726338 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.726340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.726340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.726344 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.726361 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.726365 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.726418 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.726524 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.726540 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:27:21.726809 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.724242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41981 (local address 127.6.116.129:41117)
0929 03:27:21.724420 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.724449 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.724898 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.725127 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.725133 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.725146 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.726159 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.726159 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.726184 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.726211 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.726428 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.726433 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.726436 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.726518 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.726577 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260929 03:27:21.727294 11444 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41981
I20260929 03:27:21.727363 11444 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:21.727520 11444 heartbeater.cc:511] Master 127.6.116.190:41981 requested a full tablet report, sending...
I20260929 03:27:21.728003 11256 ts_manager.cc:194] Registered new tserver with Master: aacf3f8cd3fd4fe48c780628fb4f5d67 (127.6.116.129:36671)
I20260929 03:27:21.732796 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11316
I20260929 03:27:21.732908 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:21.733103 6610 external_mini_cluster.cc:1428] Setting key f514d544e4cf6c1dbce21594b5f59c8e
I20260929 03:27:21.733922 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--builtin_ntp_servers=127.6.116.148:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 03:27:21.812098 11447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:21.812300 11447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:21.812335 11447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:21.813419 11447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:21.813478 11447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:21.813527 11447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:21.814836 11447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:21.815099 11447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:21.815367 11447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:21.820178 11452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.820266 11447 server_base.cc:1034] running on GCE node
W20260929 03:27:21.820185 11455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.820185 11453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.820569 11447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:21.820762 11447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:21.821913 11447 hybrid_clock.cc:648] HybridClock initialized: now 1790652441821891 us; error 36 us; skew 500 ppm
I20260929 03:27:21.823089 11447 webserver.cc:460] Webserver started at http://127.6.116.130:45875/ using document root <none> and password file <none>
I20260929 03:27:21.823289 11447 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:21.823330 11447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:21.823414 11447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:21.824373 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "854b402095374201ad02c6001f874b5d"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "30b16b428114efd451f0577e03ae848d"
server_key_iv: "aa9f27d5c45b3778477622ee8b1b1cb1"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.824656 11447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "854b402095374201ad02c6001f874b5d"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "30b16b428114efd451f0577e03ae848d"
server_key_iv: "aa9f27d5c45b3778477622ee8b1b1cb1"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.825814 11447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:21.826438 11461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.826609 11447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:21.826696 11447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "854b402095374201ad02c6001f874b5d"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "30b16b428114efd451f0577e03ae848d"
server_key_iv: "aa9f27d5c45b3778477622ee8b1b1cb1"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.826776 11447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:21.829983 11447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:21.830229 11447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:21.830363 11447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:21.830566 11447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:21.830847 11447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:21.830900 11447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.830936 11447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:21.830964 11447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.836455 11447 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:35491
I20260929 03:27:21.836491 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:35491 every 8 connection(s)
I20260929 03:27:21.836774 11447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:21.838279 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11447
I20260929 03:27:21.838392 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:21.838577 6610 external_mini_cluster.cc:1428] Setting key 1a9b4168ab3ec5fe7bda7d542984aea7
I20260929 03:27:21.839493 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--builtin_ntp_servers=127.6.116.148:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 03:27:21.839756 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.837420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55119 (local address 127.6.116.190:41981)
0929 03:27:21.837460 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.837462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.837692 (+ 230us) server_negotiation.cc:407] Connection header received
0929 03:27:21.838159 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.838161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.838167 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.838223 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.839516 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.839517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.839517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.839522 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.839536 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.839540 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.839597 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.839674 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.839688 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:21.839938 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.837403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41981 (local address 127.6.116.130:55119)
0929 03:27:21.837584 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.837612 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.838039 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.838285 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.838291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.838303 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.839371 (+ 1068us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.839371 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.839397 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.839423 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.839600 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.839604 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.839607 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.839681 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.839735 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260929 03:27:21.840368 11575 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41981
I20260929 03:27:21.840459 11575 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:21.840600 11575 heartbeater.cc:511] Master 127.6.116.190:41981 requested a full tablet report, sending...
I20260929 03:27:21.840955 11256 ts_manager.cc:194] Registered new tserver with Master: 854b402095374201ad02c6001f874b5d (127.6.116.130:35491)
W20260929 03:27:21.919975 11578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:21.920213 11578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:21.920253 11578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:21.921527 11578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:21.921597 11578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:21.921646 11578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:21.923128 11578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41981
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:21.923414 11578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:21.923725 11578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:21.928934 11586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.928982 11583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:21.928946 11584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:21.929318 11578 server_base.cc:1034] running on GCE node
I20260929 03:27:21.929530 11578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:21.929737 11578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:21.930882 11578 hybrid_clock.cc:648] HybridClock initialized: now 1790652441930866 us; error 29 us; skew 500 ppm
I20260929 03:27:21.932274 11578 webserver.cc:460] Webserver started at http://127.6.116.131:46677/ using document root <none> and password file <none>
I20260929 03:27:21.932550 11578 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:21.932605 11578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:21.932798 11578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:21.933866 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "6ff8fb2302d949dcaf510a52094987ca"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "a7a91bbc22442c3322b5ed35ae2294e8"
server_key_iv: "28d855917beb0edb48fa60613c6d4f7f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.934232 11578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "6ff8fb2302d949dcaf510a52094987ca"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "a7a91bbc22442c3322b5ed35ae2294e8"
server_key_iv: "28d855917beb0edb48fa60613c6d4f7f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.935768 11578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260929 03:27:21.936513 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.936679 11578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:21.936785 11578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "6ff8fb2302d949dcaf510a52094987ca"
format_stamp: "Formatted at 2026-09-29 03:27:21 on dist-test-slave-z2zg"
server_key: "a7a91bbc22442c3322b5ed35ae2294e8"
server_key_iv: "28d855917beb0edb48fa60613c6d4f7f"
server_key_version: "encryptionkey@0"
I20260929 03:27:21.936874 11578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:21.940536 11578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:21.940830 11578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:21.940996 11578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:21.941236 11578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:21.941609 11578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:21.941675 11578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.941722 11578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:21.941756 11578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:21.948868 11578 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:33775
I20260929 03:27:21.948897 11705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:33775 every 8 connection(s)
I20260929 03:27:21.949256 11578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:21.952507 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.949990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:58059 (local address 127.6.116.190:41981)
0929 03:27:21.950086 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.950089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.950274 (+ 185us) server_negotiation.cc:407] Connection header received
0929 03:27:21.950775 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.950776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.950784 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.950879 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.952175 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.952177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.952177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.952183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.952201 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.952206 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.952303 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.952398 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.952415 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 03:27:21.952694 11708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.949918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41981 (local address 127.6.116.131:58059)
0929 03:27:21.950142 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.950187 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.950660 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.950950 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.950955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.950964 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.951959 (+ 995us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.951959 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.952010 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.952038 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.952303 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.952310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.952313 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.952408 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.952470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260929 03:27:21.953166 11706 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41981
I20260929 03:27:21.953255 11706 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:21.953387 11706 heartbeater.cc:511] Master 127.6.116.190:41981 requested a full tablet report, sending...
I20260929 03:27:21.953722 11256 ts_manager.cc:194] Registered new tserver with Master: 6ff8fb2302d949dcaf510a52094987ca (127.6.116.131:33775)
I20260929 03:27:21.954145 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11578
I20260929 03:27:21.954254 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:21.954425 6610 external_mini_cluster.cc:1428] Setting key 8d833196086e0619089fc71f8408bec2
I20260929 03:27:21.955291 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:27:21.957680 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.956558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.6.116.190:41981)
0929 03:27:21.956619 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.956621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.956695 (+ 74us) server_negotiation.cc:407] Connection header received
0929 03:27:21.956785 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.956787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.956791 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.956857 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.957410 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.957412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.957412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.957417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.957430 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.957435 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.957512 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.957586 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.957604 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:27:21.959235 11256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:21.960884 11256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:21.964751 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.963384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36671 (local address 127.0.0.1:53706)
0929 03:27:21.963519 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.963531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.963587 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.963791 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.963796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.963802 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.964258 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.964259 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.964269 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.964284 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.964524 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.964534 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.964537 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.964649 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.964664 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260929 03:27:21.964776 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.963207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.6.116.129:36671)
0929 03:27:21.963559 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.963565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.963579 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:27:21.963682 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.963691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.963703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.963777 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.964361 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.964367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.964367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.964371 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.964402 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.964425 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.964489 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.964660 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.964673 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":91,"threads_started":1}
I20260929 03:27:21.965639 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.963210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:33775 (local address 127.0.0.1:46712)
0929 03:27:21.963326 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.963364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.963415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.964399 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.964400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.964403 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.965066 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.965066 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.965072 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.965084 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.965495 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.965495 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.965496 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.965544 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.965557 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260929 03:27:21.965540 11378 tablet_service.cc:1467] Processing CreateTablet for tablet e2e86c5e72e24637829ea393bae99f0d (DEFAULT_TABLE table=test-table [id=3b8dbe4e87084fa7a2389dabdfece28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.965695 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.963281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.6.116.131:33775)
0929 03:27:21.964292 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.964295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.964305 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:21.964324 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.964326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.964332 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.964396 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.965313 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.965316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.965317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.965321 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.965351 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.965365 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.965436 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.965584 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.965603 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":80,"threads_started":1}
I20260929 03:27:21.965894 11378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e86c5e72e24637829ea393bae99f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.966040 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.963127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.6.116.130:35491)
0929 03:27:21.963669 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:21.963673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:21.963683 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:21.963711 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:21.963713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:21.963723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:21.963807 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:21.965666 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:21.965670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:21.965670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:21.965674 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:21.965706 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:21.965722 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:21.965800 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:21.965921 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:27:21.965941 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":87,"threads_started":1}
I20260929 03:27:21.965986 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:21.962986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:35491 (local address 127.0.0.1:49754)
0929 03:27:21.963326 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:21.963364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:21.963415 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:21.964963 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:21.964964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:21.964967 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:21.965411 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:21.965412 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:21.965417 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:21.965428 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:21.965833 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:21.965834 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:21.965835 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:21.965891 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:27:21.965908 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":89,"threads_started":1}
I20260929 03:27:21.966980 11640 tablet_service.cc:1467] Processing CreateTablet for tablet e2e86c5e72e24637829ea393bae99f0d (DEFAULT_TABLE table=test-table [id=3b8dbe4e87084fa7a2389dabdfece28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.967018 11509 tablet_service.cc:1467] Processing CreateTablet for tablet e2e86c5e72e24637829ea393bae99f0d (DEFAULT_TABLE table=test-table [id=3b8dbe4e87084fa7a2389dabdfece28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:21.967365 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e86c5e72e24637829ea393bae99f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.967365 11509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2e86c5e72e24637829ea393bae99f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:21.968782 11720 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Bootstrap starting.
I20260929 03:27:21.969285 11720 tablet_bootstrap.cc:654] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.969489 11720 log.cc:824] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.970078 11720 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: No bootstrap required, opened a new log
I20260929 03:27:21.970118 11720 ts_tablet_manager.cc:1397] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:21.970836 11723 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Bootstrap starting.
I20260929 03:27:21.971191 11722 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Bootstrap starting.
I20260929 03:27:21.971760 11723 tablet_bootstrap.cc:654] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.971783 11722 tablet_bootstrap.cc:654] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:21.972074 11723 log.cc:824] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.972096 11722 log.cc:824] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:21.972834 11723 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: No bootstrap required, opened a new log
I20260929 03:27:21.972838 11722 tablet_bootstrap.cc:492] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: No bootstrap required, opened a new log
I20260929 03:27:21.972750 11720 raft_consensus.cc:348] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.972898 11720 raft_consensus.cc:374] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.972918 11722 ts_tablet_manager.cc:1397] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:21.972922 11723 ts_tablet_manager.cc:1397] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:21.972929 11720 raft_consensus.cc:729] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aacf3f8cd3fd4fe48c780628fb4f5d67, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.973065 11720 consensus_queue.cc:260] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.973331 11720 ts_tablet_manager.cc:1428] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:21.973374 11444 heartbeater.cc:503] Master 127.6.116.190:41981 was elected leader, sending a full tablet report...
W20260929 03:27:21.974167 11445 tablet.cc:2369] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:21.975359 11722 raft_consensus.cc:348] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.975472 11722 raft_consensus.cc:374] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.975492 11722 raft_consensus.cc:729] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 854b402095374201ad02c6001f874b5d, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.975596 11722 consensus_queue.cc:260] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.975842 11722 ts_tablet_manager.cc:1428] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:27:21.975739 11723 raft_consensus.cc:348] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.975885 11575 heartbeater.cc:503] Master 127.6.116.190:41981 was elected leader, sending a full tablet report...
I20260929 03:27:21.975909 11723 raft_consensus.cc:374] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:21.975948 11723 raft_consensus.cc:729] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ff8fb2302d949dcaf510a52094987ca, State: Initialized, Role: FOLLOWER
I20260929 03:27:21.976063 11723 consensus_queue.cc:260] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:21.976366 11723 ts_tablet_manager.cc:1428] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 03:27:21.976399 11706 heartbeater.cc:503] Master 127.6.116.190:41981 was elected leader, sending a full tablet report...
W20260929 03:27:22.087276 11576 tablet.cc:2369] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:22.180498 11727 raft_consensus.cc:482] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:22.180613 11727 raft_consensus.cc:504] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:22.181080 11727 leader_election.cc:288] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ff8fb2302d949dcaf510a52094987ca (127.6.116.131:33775), aacf3f8cd3fd4fe48c780628fb4f5d67 (127.6.116.129:36671)
I20260929 03:27:22.182461 11718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.181381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:59713 (local address 127.6.116.131:33775)
0929 03:27:22.181426 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.181427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.181436 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:22.181468 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.181468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.181472 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.181567 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.182192 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.182194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.182194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.182198 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.182214 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.182221 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.182297 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.182365 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.182379 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:22.182466 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.181255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:33775 (local address 127.6.116.130:59713)
0929 03:27:22.181314 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.181329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:22.181371 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:22.181597 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:22.181598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:22.181601 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:22.182083 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:22.182084 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:22.182091 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:22.182107 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:22.182307 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:22.182308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:22.182309 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:22.182350 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 03:27:22.182363 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:27:22.182533 11717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.181325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:45703 (local address 127.6.116.129:36671)
0929 03:27:22.181368 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.181371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.181548 (+ 177us) server_negotiation.cc:407] Connection header received
0929 03:27:22.181648 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.181649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.181654 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.181691 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.182301 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.182303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.182303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.182307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.182326 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.182329 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.182398 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.182450 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.182462 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:27:22.182578 11729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.181259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36671 (local address 127.6.116.130:45703)
0929 03:27:22.181467 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.181474 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:22.181536 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:22.181714 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:22.181715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:22.181717 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:22.182206 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:22.182207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:22.182211 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:22.182218 (+ 7us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:22.182405 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:22.182406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:22.182406 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 03:27:22.182452 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:27:22.182462 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":65,"thread_start_us":92,"threads_started":1}
I20260929 03:27:22.182927 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e86c5e72e24637829ea393bae99f0d" candidate_uuid: "854b402095374201ad02c6001f874b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" is_pre_election: true
I20260929 03:27:22.183087 11398 raft_consensus.cc:2454] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854b402095374201ad02c6001f874b5d in term 0.
I20260929 03:27:22.183318 11463 leader_election.cc:302] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854b402095374201ad02c6001f874b5d, aacf3f8cd3fd4fe48c780628fb4f5d67; no voters:
I20260929 03:27:22.183313 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e86c5e72e24637829ea393bae99f0d" candidate_uuid: "854b402095374201ad02c6001f874b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff8fb2302d949dcaf510a52094987ca" is_pre_election: true
I20260929 03:27:22.183446 11660 raft_consensus.cc:2454] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 854b402095374201ad02c6001f874b5d in term 0.
I20260929 03:27:22.183444 11727 raft_consensus.cc:2781] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:22.183512 11727 raft_consensus.cc:482] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:22.183539 11727 raft_consensus.cc:3037] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.184240 11727 raft_consensus.cc:504] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:22.184373 11727 leader_election.cc:288] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [CANDIDATE]: Term 1 election: Requested vote from peers 6ff8fb2302d949dcaf510a52094987ca (127.6.116.131:33775), aacf3f8cd3fd4fe48c780628fb4f5d67 (127.6.116.129:36671)
I20260929 03:27:22.184516 11660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e86c5e72e24637829ea393bae99f0d" candidate_uuid: "854b402095374201ad02c6001f874b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ff8fb2302d949dcaf510a52094987ca"
I20260929 03:27:22.184557 11398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2e86c5e72e24637829ea393bae99f0d" candidate_uuid: "854b402095374201ad02c6001f874b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67"
I20260929 03:27:22.184602 11660 raft_consensus.cc:3037] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.184626 11398 raft_consensus.cc:3037] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.185465 11398 raft_consensus.cc:2454] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854b402095374201ad02c6001f874b5d in term 1.
I20260929 03:27:22.185465 11660 raft_consensus.cc:2454] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 854b402095374201ad02c6001f874b5d in term 1.
I20260929 03:27:22.185616 11463 leader_election.cc:302] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 854b402095374201ad02c6001f874b5d, aacf3f8cd3fd4fe48c780628fb4f5d67; no voters:
I20260929 03:27:22.185724 11727 raft_consensus.cc:2781] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:22.185894 11727 raft_consensus.cc:686] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 1 LEADER]: Becoming Leader. State: Replica: 854b402095374201ad02c6001f874b5d, State: Running, Role: LEADER
I20260929 03:27:22.186022 11727 consensus_queue.cc:237] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } }
I20260929 03:27:22.186784 11256 catalog_manager.cc:5495] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 854b402095374201ad02c6001f874b5d (127.6.116.130). New cstate: current_term: 1 leader_uuid: "854b402095374201ad02c6001f874b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "854b402095374201ad02c6001f874b5d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:22.189929 11715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.189150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.6.116.130:35491)
0929 03:27:22.189209 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.189210 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.189217 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:22.189263 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.189264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.189272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.189312 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.189717 (+ 405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.189718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.189718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.189721 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.189732 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.189735 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.189782 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.189814 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.189823 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:27:22.191337 11398 raft_consensus.cc:1261] T e2e86c5e72e24637829ea393bae99f0d P aacf3f8cd3fd4fe48c780628fb4f5d67 [term 1 FOLLOWER]: Refusing update from remote peer 854b402095374201ad02c6001f874b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:22.191370 11660 raft_consensus.cc:1261] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 1 FOLLOWER]: Refusing update from remote peer 854b402095374201ad02c6001f874b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:22.191637 11727 consensus_queue.cc:1035] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "aacf3f8cd3fd4fe48c780628fb4f5d67" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:22.191751 11730 consensus_queue.cc:1035] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ff8fb2302d949dcaf510a52094987ca" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:22.194101 11734 mvcc.cc:205] Tried to move back new op lower bound from 7334512403213516800 to 7334512403194523648. Current Snapshot: MvccSnapshot[applied={T|T < 7334512403213516800}]
I20260929 03:27:22.196236 11256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:22.196313 11256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:22.196971 11256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 049bee76d5bf4bf9bbcef04848881ddf: Sending DeleteTablet for 3 replicas of tablet e2e86c5e72e24637829ea393bae99f0d
I20260929 03:27:22.197264 11378 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e86c5e72e24637829ea393bae99f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:22 UTC) from {username='slave'} at 127.0.0.1:53706
I20260929 03:27:22.197290 11640 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e86c5e72e24637829ea393bae99f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:22 UTC) from {username='slave'} at 127.0.0.1:46712
I20260929 03:27:22.197479 11509 tablet_service.cc:1514] Processing DeleteTablet for tablet e2e86c5e72e24637829ea393bae99f0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:22 UTC) from {username='slave'} at 127.0.0.1:49754
I20260929 03:27:22.197546 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11316
I20260929 03:27:22.197669 11743 tablet_replica.cc:330] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: stopping tablet replica
I20260929 03:27:22.197755 11743 raft_consensus.cc:2229] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:22.197842 11743 raft_consensus.cc:2258] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:22.198165 11744 tablet_replica.cc:330] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: stopping tablet replica
I20260929 03:27:22.198264 11744 raft_consensus.cc:2229] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:22.198385 11744 raft_consensus.cc:2258] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:22.198835 11743 ts_tablet_manager.cc:1905] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:22.200809 11743 ts_tablet_manager.cc:1918] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:22.200882 11743 log.cc:1196] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/e2e86c5e72e24637829ea393bae99f0d
I20260929 03:27:22.201123 11743 ts_tablet_manager.cc:1939] T e2e86c5e72e24637829ea393bae99f0d P 6ff8fb2302d949dcaf510a52094987ca: Deleting consensus metadata
I20260929 03:27:22.201716 11744 ts_tablet_manager.cc:1905] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:22.201741 11240 catalog_manager.cc:4841] TS 6ff8fb2302d949dcaf510a52094987ca (127.6.116.131:33775): tablet e2e86c5e72e24637829ea393bae99f0d (table test-table [id=3b8dbe4e87084fa7a2389dabdfece28b]) successfully deleted
I20260929 03:27:22.203644 11744 ts_tablet_manager.cc:1918] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:22.203715 11744 log.cc:1196] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/e2e86c5e72e24637829ea393bae99f0d
I20260929 03:27:22.203967 11744 ts_tablet_manager.cc:1939] T e2e86c5e72e24637829ea393bae99f0d P 854b402095374201ad02c6001f874b5d: Deleting consensus metadata
I20260929 03:27:22.204326 11240 catalog_manager.cc:4841] TS 854b402095374201ad02c6001f874b5d (127.6.116.130:35491): tablet e2e86c5e72e24637829ea393bae99f0d (table test-table [id=3b8dbe4e87084fa7a2389dabdfece28b]) successfully deleted
W20260929 03:27:22.205437 11241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:36671 (error 108)
I20260929 03:27:22.205633 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11447
I20260929 03:27:22.206234 11714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.205727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36671 (local address 127.0.0.1:53708)
0929 03:27:22.206149 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.206187 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:36671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
W20260929 03:27:22.207396 11241 catalog_manager.cc:4584] TS aacf3f8cd3fd4fe48c780628fb4f5d67 (127.6.116.129:36671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2e86c5e72e24637829ea393bae99f0d: Network error: Client connection negotiation failed: client connection to 127.6.116.129:36671: connect: Connection refused (error 111)
I20260929 03:27:22.210445 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11578
I20260929 03:27:22.214659 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11225
2026-09-29T03:27:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (767 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 03:27:22.233563 6610 mini_oidc.cc:152] Starting JWKS server
I20260929 03:27:22.234548 6610 webserver.cc:460] Webserver started at https://127.0.0.1:39837/ using document root <none> and password file <none>
I20260929 03:27:22.234642 6610 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:27:22.234824 6610 webserver.cc:460] Webserver started at http://127.0.0.1:43177/ using document root <none> and password file <none>
2026-09-29T03:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:22Z Disabled control of system clock
I20260929 03:27:22.242038 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:35805
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:35805
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39837/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/testchainca.cert with env {}
W20260929 03:27:22.317940 11751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:22.318112 11751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:22.318140 11751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:22.319315 11751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:22.319353 11751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:22.319366 11751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:22.319407 11751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:22.319437 11751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:27:22.319453 11751 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:27:22.319478 11751 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39837/jwks.json
I20260929 03:27:22.320928 11751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:35805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39837/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:22.321172 11751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:22.321377 11751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:22.325870 11756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.325863 11759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.326073 11751 server_base.cc:1034] running on GCE node
W20260929 03:27:22.325865 11757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.326396 11751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:22.326669 11751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:22.327824 11751 hybrid_clock.cc:648] HybridClock initialized: now 1790652442327804 us; error 39 us; skew 500 ppm
I20260929 03:27:22.328926 11751 webserver.cc:460] Webserver started at http://127.6.116.190:44117/ using document root <none> and password file <none>
I20260929 03:27:22.329175 11751 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:22.329242 11751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:22.329370 11751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:22.330694 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "332e7f7d312b46e0b951aec64dfeab3e"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "1e2cc9657bd1dcdf206a5c853d5a1058"
server_key_iv: "032e901e9b7d562b94c14e1acc0b96bf"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.331137 11751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "332e7f7d312b46e0b951aec64dfeab3e"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "1e2cc9657bd1dcdf206a5c853d5a1058"
server_key_iv: "032e901e9b7d562b94c14e1acc0b96bf"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.332800 11751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:22.333699 11765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:22.333930 11751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:22.334026 11751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "332e7f7d312b46e0b951aec64dfeab3e"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "1e2cc9657bd1dcdf206a5c853d5a1058"
server_key_iv: "032e901e9b7d562b94c14e1acc0b96bf"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.334093 11751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:22.346590 11751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:22.346803 11751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:22.346894 11751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:22.350558 11751 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:35805
I20260929 03:27:22.350585 11817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:35805 every 8 connection(s)
I20260929 03:27:22.350884 11751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:22.351783 11818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:22.353910 11818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e: Bootstrap starting.
I20260929 03:27:22.354475 11818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:22.354770 11818 log.cc:824] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:22.355429 11818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e: No bootstrap required, opened a new log
I20260929 03:27:22.355803 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11751
I20260929 03:27:22.355902 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:22.356091 6610 external_mini_cluster.cc:1428] Setting key 3406e34f51fbf6f50a4076af17703a72
I20260929 03:27:22.357337 11818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } }
I20260929 03:27:22.357486 11818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:22.357528 11818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332e7f7d312b46e0b951aec64dfeab3e, State: Initialized, Role: FOLLOWER
I20260929 03:27:22.357688 11818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } }
I20260929 03:27:22.357772 11818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:22.357803 11818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:22.357860 11818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.358515 11818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } }
I20260929 03:27:22.358616 11818 leader_election.cc:302] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332e7f7d312b46e0b951aec64dfeab3e; no voters:
I20260929 03:27:22.358764 11818 leader_election.cc:288] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:22.358829 11823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:22.358961 11823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [term 1 LEADER]: Becoming Leader. State: Replica: 332e7f7d312b46e0b951aec64dfeab3e, State: Running, Role: LEADER
I20260929 03:27:22.358973 11818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:22.359066 11823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } }
I20260929 03:27:22.359385 11823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332e7f7d312b46e0b951aec64dfeab3e. Latest consensus state: current_term: 1 leader_uuid: "332e7f7d312b46e0b951aec64dfeab3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } } }
I20260929 03:27:22.359479 11823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.359397 11824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332e7f7d312b46e0b951aec64dfeab3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332e7f7d312b46e0b951aec64dfeab3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35805 } } }
I20260929 03:27:22.359611 11824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.359684 11830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:22.360141 11830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:22.362547 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.356560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.6.116.190:35805)
0929 03:27:22.356770 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.356778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.356809 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:22.357281 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.357287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.357339 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.357485 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.357721 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.359380 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.360549 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.361139 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.361251 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.361863 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.361866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.361867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.362080 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.362119 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.362144 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.362214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.362276 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.362350 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260929 03:27:22.363453 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:27:22.363478 11830 catalog_manager.cc:1347] Generated new cluster ID: 4e6cdc1f11cc48bb9931ccaf13cf5141
I20260929 03:27:22.363521 11830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:22.368605 11830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:22.369112 11830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:22.379346 11830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 332e7f7d312b46e0b951aec64dfeab3e: Generated new TSK 0
I20260929 03:27:22.379498 11830 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:27:22.400591 11848 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 03:27:22.400679 11848 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:27:22.400919 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.385723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.6.116.190:35805)
0929 03:27:22.385762 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.385764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.385888 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:27:22.385948 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.385949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.385960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.386034 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:27:22.386162 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.386897 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.387710 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.387995 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.388067 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.388239 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:27:22.400709 (+ 12470us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:27:22.400860 (+ 151us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 03:27:22.400967 11822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 03:27:22.401247 11847 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:22.385648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:35805 (local address 127.0.0.1:35992)
0929 03:27:22.385816 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.385828 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:22.385885 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:22.386043 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:22.386044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:22.386045 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:27:22.386102 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.386103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.386999 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.386999 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.387635 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.387638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.388071 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.388072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.388131 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.388149 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:27:22.400862 (+ 12713us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 03:27:22.400945 (+ 83us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.116.190:35805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":95,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.116.190:35805: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39837/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 03:27:22.401701 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11751
2026-09-29T03:27:22Z chronyd exiting
I20260929 03:27:22.437016 6610 test_util.cc:175] -----------------------------------------------
I20260929 03:27:22.437099 6610 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790652420563207-6610-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 03:27:22.437878 6610 mini_oidc.cc:152] Starting JWKS server
I20260929 03:27:22.438534 6610 webserver.cc:460] Webserver started at https://127.0.0.1:34731/ using document root <none> and password file <none>
I20260929 03:27:22.438601 6610 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:27:22.438762 6610 webserver.cc:460] Webserver started at http://127.0.0.1:39263/ using document root <none> and password file <none>
2026-09-29T03:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:22Z Disabled control of system clock
I20260929 03:27:22.446429 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42717
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:35501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:42717
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/testchainca.cert with env {}
W20260929 03:27:22.523211 11854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:22.523447 11854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:22.523470 11854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:22.524564 11854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:22.524610 11854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:22.524622 11854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:22.524634 11854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:22.524647 11854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:27:22.524659 11854 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:27:22.524670 11854 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34731/jwks.json
I20260929 03:27:22.526062 11854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:42717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:22.526254 11854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:22.526397 11854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:22.530782 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.530772 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.530788 11854 server_base.cc:1034] running on GCE node
W20260929 03:27:22.530771 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.531174 11854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:22.531469 11854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:22.532611 11854 hybrid_clock.cc:648] HybridClock initialized: now 1790652442532601 us; error 25 us; skew 500 ppm
I20260929 03:27:22.533805 11854 webserver.cc:460] Webserver started at http://127.6.116.190:37513/ using document root <none> and password file <none>
I20260929 03:27:22.534039 11854 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:22.534083 11854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:22.534246 11854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:22.535260 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "c34d84839c3e484397db2c1d8c002b6b"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "e1e05db534d6f13d17c97762307ebd0b"
server_key_iv: "5ac7c30b7df744d42e1da52f4f93078f"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.535593 11854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "c34d84839c3e484397db2c1d8c002b6b"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "e1e05db534d6f13d17c97762307ebd0b"
server_key_iv: "5ac7c30b7df744d42e1da52f4f93078f"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.536783 11854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:22.537462 11868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:22.537626 11854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:22.537720 11854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "c34d84839c3e484397db2c1d8c002b6b"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "e1e05db534d6f13d17c97762307ebd0b"
server_key_iv: "5ac7c30b7df744d42e1da52f4f93078f"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.537799 11854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:22.545502 11854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:22.545765 11854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:22.545899 11854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:22.549507 11854 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:42717
I20260929 03:27:22.549537 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:42717 every 8 connection(s)
I20260929 03:27:22.549823 11854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:22.550241 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11854
I20260929 03:27:22.550354 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:22.550529 6610 external_mini_cluster.cc:1428] Setting key cbca779f1efcdb173de35d481a549721
I20260929 03:27:22.550917 11921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:22.553783 11921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b: Bootstrap starting.
I20260929 03:27:22.554430 11921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:22.554756 11921 log.cc:824] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:22.555585 11921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b: No bootstrap required, opened a new log
I20260929 03:27:22.555894 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.550978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.6.116.190:42717)
0929 03:27:22.551195 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.551207 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.551228 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:27:22.551568 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.551574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.551630 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.551728 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.551957 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.552795 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.553914 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.554520 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.554648 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.555223 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.555226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.555226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.555427 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.555465 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.555478 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.555548 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.555612 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.555687 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":101,"threads_started":1}
I20260929 03:27:22.556876 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:27:22.557915 11921 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } }
I20260929 03:27:22.558054 11921 raft_consensus.cc:374] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:22.558079 11921 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34d84839c3e484397db2c1d8c002b6b, State: Initialized, Role: FOLLOWER
I20260929 03:27:22.558156 11921 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } }
I20260929 03:27:22.558204 11921 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:22.558224 11921 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:22.558249 11921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.558714 11921 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } }
I20260929 03:27:22.558801 11921 leader_election.cc:302] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34d84839c3e484397db2c1d8c002b6b; no voters:
I20260929 03:27:22.558903 11921 leader_election.cc:288] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:22.558955 11926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:22.559111 11926 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [term 1 LEADER]: Becoming Leader. State: Replica: c34d84839c3e484397db2c1d8c002b6b, State: Running, Role: LEADER
I20260929 03:27:22.559149 11921 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:22.559198 11926 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } }
I20260929 03:27:22.559522 11927 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c34d84839c3e484397db2c1d8c002b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } } }
I20260929 03:27:22.559527 11928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34d84839c3e484397db2c1d8c002b6b. Latest consensus state: current_term: 1 leader_uuid: "c34d84839c3e484397db2c1d8c002b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34d84839c3e484397db2c1d8c002b6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42717 } } }
I20260929 03:27:22.559702 11928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.559686 11927 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.560101 11935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:22.560565 11935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:22.561887 11935 catalog_manager.cc:1347] Generated new cluster ID: ca612272ab79496c8e8141439a025328
I20260929 03:27:22.561937 11935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:22.575877 11935 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:22.576504 11935 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:22.592319 11935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34d84839c3e484397db2c1d8c002b6b: Generated new TSK 0
I20260929 03:27:22.592483 11935 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 03:27:22.612498 11951 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 03:27:22.612566 11951 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 03:27:22.612849 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.598815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.6.116.190:42717)
0929 03:27:22.598873 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.598875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.599025 (+ 150us) server_negotiation.cc:407] Connection header received
0929 03:27:22.599075 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.599076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.599088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.599150 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:27:22.599270 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.600009 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.600712 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.601018 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.601086 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.601228 (+ 142us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:27:22.612663 (+ 11435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:27:22.612788 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 03:27:22.612897 11924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 03:27:22.612916 11950 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:22.598754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:42717 (local address 127.0.0.1:57086)
0929 03:27:22.598921 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.598929 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:22.598985 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:22.599162 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:22.599163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:22.599164 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:27:22.599205 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.599207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.600079 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.600080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.600631 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.600632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.601091 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.601091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.601134 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.601138 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:27:22.612750 (+ 11612us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 03:27:22.612831 (+ 81us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.116.190:42717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34731/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260929 03:27:22.613199 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11854
2026-09-29T03:27:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 03:27:22.650751 6610 mini_oidc.cc:152] Starting JWKS server
I20260929 03:27:22.651469 6610 webserver.cc:460] Webserver started at https://127.0.0.1:34311/ using document root <none> and password file <none>
I20260929 03:27:22.651535 6610 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 03:27:22.651690 6610 webserver.cc:460] Webserver started at http://127.0.0.1:38891/ using document root <none> and password file <none>
2026-09-29T03:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:22Z Disabled control of system clock
I20260929 03:27:22.658864 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40511
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:38881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:40511
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34311/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:27:22.738986 11957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:22.739187 11957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:22.739207 11957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:22.740271 11957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:22.740304 11957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:22.740317 11957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:22.740330 11957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:22.740342 11957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:27:22.740355 11957 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:27:22.740370 11957 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34311/jwks.json
I20260929 03:27:22.741616 11957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:40511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34311/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:22.741789 11957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:22.742000 11957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:22.746688 11962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.746696 11965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.746685 11963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.746758 11957 server_base.cc:1034] running on GCE node
I20260929 03:27:22.747182 11957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:22.747459 11957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:22.748615 11957 hybrid_clock.cc:648] HybridClock initialized: now 1790652442748595 us; error 32 us; skew 500 ppm
I20260929 03:27:22.749728 11957 webserver.cc:460] Webserver started at http://127.6.116.190:44007/ using document root <none> and password file <none>
I20260929 03:27:22.749958 11957 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:22.750025 11957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:22.750159 11957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:22.751231 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "690e1ed7c7d547c2b9066b2ccad166e2"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "d81c6debc1e0934d3a572961d155d5d4"
server_key_iv: "60c8dcbd361a8c776a8e772bb0e5a6a9"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.751573 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "690e1ed7c7d547c2b9066b2ccad166e2"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "d81c6debc1e0934d3a572961d155d5d4"
server_key_iv: "60c8dcbd361a8c776a8e772bb0e5a6a9"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.752847 11957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:22.753542 11971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:22.753711 11957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:22.753806 11957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "690e1ed7c7d547c2b9066b2ccad166e2"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "d81c6debc1e0934d3a572961d155d5d4"
server_key_iv: "60c8dcbd361a8c776a8e772bb0e5a6a9"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.753885 11957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:22.770393 11957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:22.770658 11957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:22.770817 11957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:22.774580 11957 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:40511
I20260929 03:27:22.774606 12023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:40511 every 8 connection(s)
I20260929 03:27:22.774942 11957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:22.775563 12024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:22.778129 12024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: Bootstrap starting.
I20260929 03:27:22.778656 12024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:22.778921 12024 log.cc:824] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:22.779520 12024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: No bootstrap required, opened a new log
I20260929 03:27:22.780918 12024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } }
I20260929 03:27:22.781041 12024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:22.781064 12024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690e1ed7c7d547c2b9066b2ccad166e2, State: Initialized, Role: FOLLOWER
I20260929 03:27:22.781174 12024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } }
I20260929 03:27:22.781265 12024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:22.781306 12024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:22.781397 12024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.781929 12024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } }
I20260929 03:27:22.782042 12024 leader_election.cc:302] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 690e1ed7c7d547c2b9066b2ccad166e2; no voters:
I20260929 03:27:22.782214 12024 leader_election.cc:288] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:22.782255 12027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:22.782423 12027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [term 1 LEADER]: Becoming Leader. State: Replica: 690e1ed7c7d547c2b9066b2ccad166e2, State: Running, Role: LEADER
I20260929 03:27:22.782461 12024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:22.782521 12027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } }
I20260929 03:27:22.782898 12028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } } }
I20260929 03:27:22.782990 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 11957
I20260929 03:27:22.783118 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:22.783330 6610 external_mini_cluster.cc:1428] Setting key f23647c1ebcab967107d034bfb7ffffe
I20260929 03:27:22.783423 12028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.783332 12029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690e1ed7c7d547c2b9066b2ccad166e2. Latest consensus state: current_term: 1 leader_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690e1ed7c7d547c2b9066b2ccad166e2" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40511 } } }
I20260929 03:27:22.783834 12029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2 [sys.catalog]: This master's current role is: LEADER
W20260929 03:27:22.784212 12044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:22.784260 12044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:22.784427 12040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:22.784593 12040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:22.786067 12040 catalog_manager.cc:1347] Generated new cluster ID: de43191d26ed4c0c9bb7bbc1ab775f3d
I20260929 03:27:22.786125 12040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:22.788823 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.783878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54208 (local address 127.6.116.190:40511)
0929 03:27:22.784233 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.784239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.784294 (+ 55us) server_negotiation.cc:407] Connection header received
0929 03:27:22.784404 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.784409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.784458 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.784618 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.784879 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.786095 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.786900 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.787391 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.787529 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.788191 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.788194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.788194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.788362 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.788394 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.788403 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.788474 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.788545 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.788623 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":76,"threads_started":1}
I20260929 03:27:22.789760 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:27:22.796278 12040 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:22.796815 12040 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:22.802500 12040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 690e1ed7c7d547c2b9066b2ccad166e2: Generated new TSK 0
I20260929 03:27:22.802670 12040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:22.811080 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.807349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.6.116.190:40511)
0929 03:27:22.807400 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.807402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.807534 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:27:22.807569 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.807569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.807581 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.807661 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:27:22.807777 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.808555 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.809487 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.809821 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.809889 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.810035 (+ 146us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:27:22.810875 (+ 840us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:27:22.810997 (+ 122us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:27:22.811137 12046 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54222: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 03:27:22.812915 12054 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:22.807277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40511 (local address 127.0.0.1:54222)
0929 03:27:22.807423 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:22.807434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:22.807524 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:22.807660 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:22.807660 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:22.807661 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:27:22.807711 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.807713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.808686 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.808687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.809390 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:22.809392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.809898 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.809898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:22.809938 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.809942 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:27:22.810973 (+ 1031us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 03:27:22.812823 (+ 1850us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.116.190:40511: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34311/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1675,"thread_start_us":76,"threads_started":1}
I20260929 03:27:22.813045 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 11957
E20260929 03:27:22.823364 12055 webserver.cc:533] Webserver: sslize failed:
E20260929 03:27:22.823443 12055 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T03:27:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T03:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:22Z Disabled control of system clock
I20260929 03:27:22.858363 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36999
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:38177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:36999
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/testchainca.cert with env {}
W20260929 03:27:22.937314 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:22.937486 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:22.937506 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:22.938668 12059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:22.938709 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:22.938721 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:22.938771 12059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:22.938802 12059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 03:27:22.938822 12059 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 03:27:22.938848 12059 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 03:27:22.940361 12059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:36999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:22.940599 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:22.940801 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:22.945304 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.945276 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:22.945284 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:22.945456 12059 server_base.cc:1034] running on GCE node
I20260929 03:27:22.945760 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:22.946023 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:22.947161 12059 hybrid_clock.cc:648] HybridClock initialized: now 1790652442947132 us; error 51 us; skew 500 ppm
I20260929 03:27:22.948248 12059 webserver.cc:460] Webserver started at http://127.6.116.190:42093/ using document root <none> and password file <none>
I20260929 03:27:22.948472 12059 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:22.948514 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:22.948702 12059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:22.949647 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "ee728b973d00482dbc368f497415792d"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "2f7b06d72afde87de97202bdefdf3e2b"
server_key_iv: "b71eb63cfb49368c3bdb2f3787283dc2"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.949982 12059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "ee728b973d00482dbc368f497415792d"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "2f7b06d72afde87de97202bdefdf3e2b"
server_key_iv: "b71eb63cfb49368c3bdb2f3787283dc2"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.951269 12059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:27:22.951979 12073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:22.952121 12059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:22.952200 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "ee728b973d00482dbc368f497415792d"
format_stamp: "Formatted at 2026-09-29 03:27:22 on dist-test-slave-z2zg"
server_key: "2f7b06d72afde87de97202bdefdf3e2b"
server_key_iv: "b71eb63cfb49368c3bdb2f3787283dc2"
server_key_version: "encryptionkey@0"
I20260929 03:27:22.952250 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:22.963230 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:22.963426 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:22.963515 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:22.967239 12059 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:36999
I20260929 03:27:22.967279 12125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:36999 every 8 connection(s)
I20260929 03:27:22.967571 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:22.968310 12126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:22.970238 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: Bootstrap starting.
I20260929 03:27:22.970758 12126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:22.971032 12126 log.cc:824] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:22.971642 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: No bootstrap required, opened a new log
I20260929 03:27:22.972173 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12059
I20260929 03:27:22.972285 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:22.972453 6610 external_mini_cluster.cc:1428] Setting key 05512cfd00d7c257c3582897c5f51401
I20260929 03:27:22.973593 12126 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } }
I20260929 03:27:22.973744 12126 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:22.973790 12126 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee728b973d00482dbc368f497415792d, State: Initialized, Role: FOLLOWER
I20260929 03:27:22.973912 12126 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } }
I20260929 03:27:22.973999 12126 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:22.974035 12126 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:22.974083 12126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:22.974565 12126 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } }
I20260929 03:27:22.974690 12126 leader_election.cc:302] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee728b973d00482dbc368f497415792d; no voters:
I20260929 03:27:22.974838 12126 leader_election.cc:288] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:22.974869 12131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:22.974965 12131 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [term 1 LEADER]: Becoming Leader. State: Replica: ee728b973d00482dbc368f497415792d, State: Running, Role: LEADER
I20260929 03:27:22.975064 12131 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } }
I20260929 03:27:22.975104 12126 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:22.975353 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee728b973d00482dbc368f497415792d. Latest consensus state: current_term: 1 leader_uuid: "ee728b973d00482dbc368f497415792d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } } }
I20260929 03:27:22.975430 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:22.975364 12132 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee728b973d00482dbc368f497415792d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee728b973d00482dbc368f497415792d" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36999 } } }
I20260929 03:27:22.975574 12132 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d [sys.catalog]: This master's current role is: LEADER
W20260929 03:27:22.976836 12147 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:22.976904 12147 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:22.976948 12136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:22.977093 12136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:22.978613 12136 catalog_manager.cc:1347] Generated new cluster ID: 5a2c79cd254f4e32923bf3eff3e2201a
I20260929 03:27:22.978674 12136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:22.978809 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:22.972912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.6.116.190:36999)
0929 03:27:22.973118 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:22.973126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:22.973157 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:22.973577 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:22.973583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:22.973643 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:22.973774 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:22.973989 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.975540 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.976582 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:22.977230 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:22.977355 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:22.978023 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:22.978027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:22.978027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:22.978266 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:22.978303 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:22.978324 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:22.978403 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:22.978468 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:27:22.978543 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260929 03:27:22.979691 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:27:23.002621 12136 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:23.003172 12136 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:23.009711 12136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee728b973d00482dbc368f497415792d: Generated new TSK 0
I20260929 03:27:23.009864 12136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:23.506142 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:23.016524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.6.116.190:36999)
0929 03:27:23.016565 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:23.016567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:23.016646 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:27:23.016707 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:23.016707 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:23.016717 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:23.016771 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0929 03:27:23.016919 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:23.017379 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:23.018517 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:23.018694 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:23.018756 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:23.018926 (+ 170us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 03:27:23.505912 (+486986us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:27:23.506074 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33536: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:27:23.506245 12130 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33536: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 03:27:23.506268 12155 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:23.016433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36999 (local address 127.0.0.1:33536)
0929 03:27:23.016576 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:23.016586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:23.016647 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:23.016772 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:23.016773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:23.016774 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0929 03:27:23.016828 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:23.016830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:23.017482 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:23.017483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:23.018409 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:23.018412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:23.018776 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:23.018777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:23.018834 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:23.018840 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 03:27:23.506062 (+487222us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 03:27:23.506181 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.116.190:36999: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260929 03:27:23.506596 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12059
2026-09-29T03:27:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (672 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:23 dist-test-slave-z2zg krb5kdc[12167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:23 dist-test-slave-z2zg krb5kdc[12167](info): set up 2 sockets
Sep 29 03:27:23 dist-test-slave-z2zg krb5kdc[12167](info): commencing operation
krb5kdc: starting...
W20260929 03:27:24.716825 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:24 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:24.821285 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:33111
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:33111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:24.897898 12180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:24.898073 12180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:24.898093 12180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:24.899241 12180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:24.899279 12180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:24.899293 12180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:24.899333 12180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:24.899358 12180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:24.900758 12180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:33111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:33111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:24.900985 12180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:24.901207 12180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:24.905474 12188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:24.905510 12186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:24.905514 12185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:24.905561 12180 server_base.cc:1034] running on GCE node
I20260929 03:27:24.905936 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:24.906184 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:24.907341 12180 hybrid_clock.cc:648] HybridClock initialized: now 1790652444907329 us; error 28 us; skew 500 ppm
Sep 29 03:27:24 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:24.912956 12180 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:24.913369 12180 webserver.cc:460] Webserver started at http://127.6.116.190:36795/ using document root <none> and password file <none>
I20260929 03:27:24.913564 12180 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:24.913632 12180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:24.913761 12180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:24.914753 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "6d61a05df7414a8fbafcb96bd8c85f94"
format_stamp: "Formatted at 2026-09-29 03:27:24 on dist-test-slave-z2zg"
server_key: "a80159f96541e993c15e0968ae5cfb72"
server_key_iv: "6472b4e42776a784fade8268e186a4bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:24.915140 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "6d61a05df7414a8fbafcb96bd8c85f94"
format_stamp: "Formatted at 2026-09-29 03:27:24 on dist-test-slave-z2zg"
server_key: "a80159f96541e993c15e0968ae5cfb72"
server_key_iv: "6472b4e42776a784fade8268e186a4bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:24.916399 12180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:24.917078 12195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:24.917248 12180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:24.917338 12180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "6d61a05df7414a8fbafcb96bd8c85f94"
format_stamp: "Formatted at 2026-09-29 03:27:24 on dist-test-slave-z2zg"
server_key: "a80159f96541e993c15e0968ae5cfb72"
server_key_iv: "6472b4e42776a784fade8268e186a4bd"
server_key_version: "encryptionkey@0"
I20260929 03:27:24.917415 12180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:24.931716 12180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:24.933239 12180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:24.933386 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:24.937135 12180 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:33111
I20260929 03:27:24.937172 12247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:33111 every 8 connection(s)
I20260929 03:27:24.937474 12180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:24.938210 12248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:24.940197 12248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94: Bootstrap starting.
I20260929 03:27:24.940730 12248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:24.941035 12248 log.cc:824] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:24.941679 12248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94: No bootstrap required, opened a new log
I20260929 03:27:24.943252 12248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } }
I20260929 03:27:24.943370 12248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:24.943391 12248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d61a05df7414a8fbafcb96bd8c85f94, State: Initialized, Role: FOLLOWER
I20260929 03:27:24.943555 12248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } }
I20260929 03:27:24.943614 12248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:24.943665 12248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:24.943719 12248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:24.944206 12248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } }
I20260929 03:27:24.944303 12248 leader_election.cc:302] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d61a05df7414a8fbafcb96bd8c85f94; no voters:
I20260929 03:27:24.944463 12248 leader_election.cc:288] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:24.944509 12251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:24.944669 12251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [term 1 LEADER]: Becoming Leader. State: Replica: 6d61a05df7414a8fbafcb96bd8c85f94, State: Running, Role: LEADER
I20260929 03:27:24.944691 12248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:24.944836 12251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } }
I20260929 03:27:24.945009 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12180
I20260929 03:27:24.945108 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:24.945329 6610 external_mini_cluster.cc:1428] Setting key 822b73d34f6bc3b9eb7423428476d158
I20260929 03:27:24.945552 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d61a05df7414a8fbafcb96bd8c85f94. Latest consensus state: current_term: 1 leader_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } } }
I20260929 03:27:24.945966 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:24.945878 12252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d61a05df7414a8fbafcb96bd8c85f94" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33111 } } }
I20260929 03:27:24.946213 12267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:24.946298 12252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:24.946873 12267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:24.949191 12267 catalog_manager.cc:1347] Generated new cluster ID: 7179f1a8f951475685586e408579e6b9
I20260929 03:27:24.949240 12267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:24 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:24.956449 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:24.946103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.6.116.190:33111)
0929 03:27:24.946272 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:24.946282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:24.946358 (+ 76us) server_negotiation.cc:407] Connection header received
0929 03:27:24.946481 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:24.946487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:24.946539 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:24.946672 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:24.947491 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:24.948775 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:24.949722 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:24.950307 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:24.950413 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:24.953037 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:24.953040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:24.953041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:24.953051 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:24.954902 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:24.955306 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:24.955309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:24.955309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:24.955416 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:24.955625 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:24.955625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:24.955625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:24.955898 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:24.956018 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:24.956144 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 03:27:24.956264 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260929 03:27:24.964696 12267 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:24.965387 12267 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:24.971503 12267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d61a05df7414a8fbafcb96bd8c85f94: Generated new TSK 0
I20260929 03:27:24.971640 12267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:24.996155 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:25.073139 12274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:25.073312 12274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:25.073334 12274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:25.074347 12274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:25.074394 12274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:25.074419 12274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:25.075809 12274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:25.075982 12274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:25.076143 12274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:25.080806 12280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:25.080811 12282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:25.080895 12274 server_base.cc:1034] running on GCE node
W20260929 03:27:25.080850 12279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:25.081207 12274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:25.081453 12274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:25.082610 12274 hybrid_clock.cc:648] HybridClock initialized: now 1790652445082588 us; error 39 us; skew 500 ppm
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.085518 12274 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:25.085984 12274 webserver.cc:460] Webserver started at http://127.6.116.129:45105/ using document root <none> and password file <none>
I20260929 03:27:25.086226 12274 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:25.086274 12274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:25.086432 12274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:25.087493 12274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "4f8aaf2de51e4088a79a23d4575c145d"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "ea67fff8cbddc424e290d10045f81bec"
server_key_iv: "af39da8f337dc872ef4a888420e84926"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.087842 12274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "4f8aaf2de51e4088a79a23d4575c145d"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "ea67fff8cbddc424e290d10045f81bec"
server_key_iv: "af39da8f337dc872ef4a888420e84926"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.089134 12274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:25.089788 12289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.089946 12274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:25.090024 12274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "4f8aaf2de51e4088a79a23d4575c145d"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "ea67fff8cbddc424e290d10045f81bec"
server_key_iv: "af39da8f337dc872ef4a888420e84926"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.090086 12274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:25.097175 12274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:25.099201 12274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:25.099330 12274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:25.099503 12274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:25.099785 12274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:25.099821 12274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.099884 12274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:25.099910 12274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.105192 12274 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:33787
I20260929 03:27:25.105250 12402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:33787 every 8 connection(s)
I20260929 03:27:25.105531 12274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:25.109869 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12274
I20260929 03:27:25.109972 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:25.110173 6610 external_mini_cluster.cc:1428] Setting key c04dd5d2e1f7ee0ec8bafb2a6fd231c6
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:25.114466 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.106294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:35631 (local address 127.6.116.190:33111)
0929 03:27:25.106361 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.106363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.106590 (+ 227us) server_negotiation.cc:407] Connection header received
0929 03:27:25.107646 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.107647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.107660 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.107710 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.108211 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.108672 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.109531 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.109714 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.109762 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.111765 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.111767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.111767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.111775 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.113129 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.113552 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.113554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.113554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.113627 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.113860 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.113861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.113861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.113967 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.114040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.114298 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.114379 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:25.114607 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.106284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:33111 (local address 127.6.116.129:35631)
0929 03:27:25.106437 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:25.106482 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:25.107556 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:25.107756 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:25.107759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:25.107980 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:25.108145 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.108148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.108739 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.108740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.109461 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.109463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.109788 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.109789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.109939 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.110423 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:25.110431 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:25.111677 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:25.113225 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.113231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.113233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.113471 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.113680 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.113680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.113681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.113805 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.114031 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:25.114037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:25.114164 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 03:27:25.114265 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:27:25.114402 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260929 03:27:25.115069 12403 heartbeater.cc:347] Connected to a master server at 127.6.116.190:33111
I20260929 03:27:25.115176 12403 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:25.115331 12403 heartbeater.cc:511] Master 127.6.116.190:33111 requested a full tablet report, sending...
I20260929 03:27:25.115830 12212 ts_manager.cc:194] Registered new tserver with Master: 4f8aaf2de51e4088a79a23d4575c145d (127.6.116.129:33787)
I20260929 03:27:25.116721 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:35631
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:25.131887 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:25.208011 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:25.208189 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:25.208210 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:25.209228 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:25.209259 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:25.209282 12408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:25.210580 12408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:25.210749 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:25.210903 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:25.215476 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:25.215482 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:25.215482 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:25.215509 12408 server_base.cc:1034] running on GCE node
I20260929 03:27:25.215891 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:25.216125 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:25.217274 12408 hybrid_clock.cc:648] HybridClock initialized: now 1790652445217255 us; error 32 us; skew 500 ppm
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.219681 12408 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:25.220038 12408 webserver.cc:460] Webserver started at http://127.6.116.130:34625/ using document root <none> and password file <none>
I20260929 03:27:25.220227 12408 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:25.220264 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:25.220400 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:25.221314 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "49fbf69d597d4719a3e5fd38e008e871"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "c96ed8f7a4799ce6c7d990b74c894982"
server_key_iv: "69ad3ae5fd757250f92fabfad6e1d58a"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.221647 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "49fbf69d597d4719a3e5fd38e008e871"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "c96ed8f7a4799ce6c7d990b74c894982"
server_key_iv: "69ad3ae5fd757250f92fabfad6e1d58a"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.222838 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:27:25.223533 12423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.223701 12408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.223781 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "49fbf69d597d4719a3e5fd38e008e871"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "c96ed8f7a4799ce6c7d990b74c894982"
server_key_iv: "69ad3ae5fd757250f92fabfad6e1d58a"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.223848 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:25.248368 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:25.249940 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:25.250083 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:25.250283 12408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:25.250553 12408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:25.250586 12408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.250604 12408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:25.250649 12408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.256098 12408 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:42913
I20260929 03:27:25.256172 12536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:42913 every 8 connection(s)
I20260929 03:27:25.256428 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:25.264336 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.257100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:53735 (local address 127.6.116.190:33111)
0929 03:27:25.257176 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.257179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.257322 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:27:25.258124 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.258125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.258136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.258184 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.258664 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.259147 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.260012 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.260189 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.260237 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.262045 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.262046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.262047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.262053 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.263220 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.263554 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.263554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.263555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.263602 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.263877 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.263877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.263877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.263967 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.264014 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.264219 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.264279 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:27:25.264490 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.257063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:33111 (local address 127.6.116.130:53735)
0929 03:27:25.257215 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:25.257246 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:25.258043 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:25.258232 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:25.258235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:25.258440 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:25.258599 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.258603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.259215 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.259216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.259945 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.259947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.260253 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.260253 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.260340 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.260722 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:25.260729 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:25.261927 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:25.263293 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.263299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.263301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.263497 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.263703 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.263704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.263704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.263816 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.264020 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:25.264024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:25.264121 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:27:25.264198 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:27:25.264304 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260929 03:27:25.264942 12537 heartbeater.cc:347] Connected to a master server at 127.6.116.190:33111
I20260929 03:27:25.265022 12537 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:25.265151 12537 heartbeater.cc:511] Master 127.6.116.190:33111 requested a full tablet report, sending...
I20260929 03:27:25.265482 12212 ts_manager.cc:194] Registered new tserver with Master: 49fbf69d597d4719a3e5fd38e008e871 (127.6.116.130:42913)
I20260929 03:27:25.265851 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:53735
I20260929 03:27:25.265899 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12408
I20260929 03:27:25.266026 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:25.266239 6610 external_mini_cluster.cc:1428] Setting key e344f2dd8e53b6ccedf3ba9d66a363a8
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:25.287757 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:25.364475 12542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:25.364645 12542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:25.364666 12542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:25.365774 12542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:25.365814 12542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:25.365862 12542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:25.367312 12542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:33111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:25.367518 12542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:25.367722 12542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:25.372445 12550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:25.372449 12547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:25.372521 12542 server_base.cc:1034] running on GCE node
W20260929 03:27:25.372463 12548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:25.372808 12542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:25.373049 12542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:25.374185 12542 hybrid_clock.cc:648] HybridClock initialized: now 1790652445374180 us; error 35 us; skew 500 ppm
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.376556 12542 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:25.376943 12542 webserver.cc:460] Webserver started at http://127.6.116.131:40527/ using document root <none> and password file <none>
I20260929 03:27:25.377163 12542 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:25.377224 12542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:25.377365 12542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:25.378401 12542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "57abf19db4a643889130526832f7f3e3"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "68899f2b2afff8c03ab9a8b0e31acea3"
server_key_iv: "56c0f2e5410788beb9117ed8cbd9b720"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.378741 12542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "57abf19db4a643889130526832f7f3e3"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "68899f2b2afff8c03ab9a8b0e31acea3"
server_key_iv: "56c0f2e5410788beb9117ed8cbd9b720"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.380017 12542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:25.380649 12557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.380795 12542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:25.380887 12542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "57abf19db4a643889130526832f7f3e3"
format_stamp: "Formatted at 2026-09-29 03:27:25 on dist-test-slave-z2zg"
server_key: "68899f2b2afff8c03ab9a8b0e31acea3"
server_key_iv: "56c0f2e5410788beb9117ed8cbd9b720"
server_key_version: "encryptionkey@0"
I20260929 03:27:25.380956 12542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:25.416574 12542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:25.418951 12542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:25.419135 12542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:25.419333 12542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:25.419600 12542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:25.419654 12542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.419687 12542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:25.419716 12542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:25.425119 12542 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:37087
I20260929 03:27:25.425186 12670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:37087 every 8 connection(s)
I20260929 03:27:25.425458 12542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:25.431752 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12542
I20260929 03:27:25.431850 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:25.432049 6610 external_mini_cluster.cc:1428] Setting key 42a3b50100d5d2ea1093829ac930e489
I20260929 03:27:25.434022 12269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.426109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:37155 (local address 127.6.116.190:33111)
0929 03:27:25.426163 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.426165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.426384 (+ 219us) server_negotiation.cc:407] Connection header received
0929 03:27:25.427402 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.427403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.427414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.427482 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.428013 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.428473 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.429339 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.429505 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.429552 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.431491 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.431492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.431492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.431499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.433034 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.433347 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.433348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.433348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.433395 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.433574 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.433574 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.433574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.433671 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.433717 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.433910 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.433962 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:27:25.434207 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.426087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:33111 (local address 127.6.116.131:37155)
0929 03:27:25.426253 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:25.426308 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:25.427279 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:25.427557 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:25.427560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:25.427785 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:25.427948 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.427951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.428541 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.428542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.429276 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:25.429277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.429584 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.429585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:25.429698 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.430167 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:25.430176 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:25.431413 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:25.433092 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.433096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.433098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.433291 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.433445 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:25.433446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:25.433446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:25.433527 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:25.433718 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:25.433722 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:25.433815 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 03:27:25.433888 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:27:25.433989 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260929 03:27:25.434602 12671 heartbeater.cc:347] Connected to a master server at 127.6.116.190:33111
I20260929 03:27:25.434706 12671 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:25.434886 12671 heartbeater.cc:511] Master 127.6.116.190:33111 requested a full tablet report, sending...
I20260929 03:27:25.435266 12212 ts_manager.cc:194] Registered new tserver with Master: 57abf19db4a643889130526832f7f3e3 (127.6.116.131:37087)
I20260929 03:27:25.435577 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:37155
I20260929 03:27:25.443677 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/test-admin.keytab.
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.458828 6610 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 03:27:25.460253 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:25.459595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57102 (local address 127.6.116.129:33787)
0929 03:27:25.459755 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.459759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.459769 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:25.459791 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.459792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.459804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.459864 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.460183 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57102: BlockingRecv error: recv got EOF from 127.0.0.1:57102 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260929 03:27:25.460340 6610 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc' of type FILE
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.461246 6610 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.462174 6610 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:25.469067 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.462611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.6.116.129:33787)
0929 03:27:25.462662 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.462664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.462708 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:27:25.462749 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.462750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.462759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.462810 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.463182 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.463864 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.464534 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.464828 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.464882 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.466775 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.466781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.466781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.466787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.468142 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.468429 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.468431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.468431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.468481 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.468620 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.468620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.468621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.468721 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.468802 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.468865 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.468919 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 03:27:25.470660 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:25.470091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57120 (local address 127.6.116.129:33787)
0929 03:27:25.470134 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.470135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.470166 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:25.470232 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.470232 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.470242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.470293 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.470616 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57120: BlockingRecv error: recv got EOF from 127.0.0.1:57120 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:27:25.470790 6610 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc)
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.471737 6610 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.472549 6610 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:25.478659 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.472979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.6.116.129:33787)
0929 03:27:25.473021 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.473022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.473057 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:27:25.473101 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.473102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.473109 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.473149 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.473460 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.473874 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.474435 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.474609 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.474657 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.476468 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.476469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.476469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.476474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.477585 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.477923 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.477924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.477924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.478013 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.478239 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.478239 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.478240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.478326 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.478392 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.478491 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.478543 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 03:27:25.480029 6610 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790652420563207-6610-0/krb5kdc/krb5cc' of type FILE
W20260929 03:27:25.480018 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:25.479516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.6.116.129:33787)
0929 03:27:25.479563 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.479564 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.479600 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:27:25.479646 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.479647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.479653 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.479695 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.479964 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57138: BlockingRecv error: recv got EOF from 127.0.0.1:57138 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.480882 6610 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:25.481738 6610 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:27:25.488336 12678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:25.482219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.6.116.129:33787)
0929 03:27:25.482266 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:25.482267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:25.482300 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:27:25.482352 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:25.482353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:25.482359 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:25.482401 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:25.482752 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.483295 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.483912 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:25.484088 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:25.484148 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:25.486087 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:25.486087 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:25.486088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:25.486094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:25.487380 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.487688 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.487688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.487689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.487739 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:25.487911 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:25.487911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:25.487912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:25.487994 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:25.488061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:25.488154 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 03:27:25.488205 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:27:25.488906 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12274
I20260929 03:27:25.494472 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12408
I20260929 03:27:25.499125 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12542
I20260929 03:27:25.503194 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12180
2026-09-29T03:27:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1828 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12691](info): set up 2 sockets
Sep 29 03:27:25 dist-test-slave-z2zg krb5kdc[12691](info): commencing operation
krb5kdc: starting...
W20260929 03:27:26.516815 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260929 03:27:26.741918 6610 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.140s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/test-user.keytab.
W20260929 03:27:26.904085 6610 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.162s user 0.000s sys 0.001s
Sep 29 03:27:26 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab.
Entry for principal oryx/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab.
I20260929 03:27:26.943842 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:46193
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab
--principal=oryx/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.019573 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.019750 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.019770 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.020902 12704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:27.020943 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.020957 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.021008 12704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:27.021039 12704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:27.022415 12704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab
--principal=oryx/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:27.022642 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.022841 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.027395 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.027385 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.027416 12704 server_base.cc:1034] running on GCE node
W20260929 03:27:27.027397 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.027886 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.028112 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.029246 12704 hybrid_clock.cc:648] HybridClock initialized: now 1790652447029228 us; error 32 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.031531 12704 init.cc:377] Logged in from keytab as oryx/127.6.116.190@KRBTEST.COM (short username oryx)
I20260929 03:27:27.031922 12704 webserver.cc:460] Webserver started at http://127.6.116.190:42429/ using document root <none> and password file <none>
I20260929 03:27:27.032101 12704 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.032142 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.032225 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.033309 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "9193e24069244623b206eb66bd093c34"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "dc0049db1f32baac8a5544a698c155af"
server_key_iv: "27d62683889c5ab5a3d1968f58d0621f"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.033640 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "9193e24069244623b206eb66bd093c34"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "dc0049db1f32baac8a5544a698c155af"
server_key_iv: "27d62683889c5ab5a3d1968f58d0621f"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.034749 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:27.035425 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.035612 12704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.035702 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "9193e24069244623b206eb66bd093c34"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "dc0049db1f32baac8a5544a698c155af"
server_key_iv: "27d62683889c5ab5a3d1968f58d0621f"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.035773 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.046903 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.048580 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.048717 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.052403 12704 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:46193
I20260929 03:27:27.052448 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:46193 every 8 connection(s)
I20260929 03:27:27.052757 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:27.053625 12772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:27.055677 12772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Bootstrap starting.
I20260929 03:27:27.056216 12772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:27.056505 12772 log.cc:824] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:27.057224 12772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: No bootstrap required, opened a new log
I20260929 03:27:27.057336 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12704
I20260929 03:27:27.057439 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:27.057638 6610 external_mini_cluster.cc:1428] Setting key f62a63f135189086a07f6e8cb2eb7f85
I20260929 03:27:27.059191 12772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } }
I20260929 03:27:27.059332 12772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:27.059369 12772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9193e24069244623b206eb66bd093c34, State: Initialized, Role: FOLLOWER
I20260929 03:27:27.059526 12772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } }
I20260929 03:27:27.059614 12772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:27.059649 12772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:27.059698 12772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:27.060598 12772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } }
I20260929 03:27:27.060678 12772 leader_election.cc:302] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9193e24069244623b206eb66bd093c34; no voters:
I20260929 03:27:27.060827 12772 leader_election.cc:288] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:27.060876 12777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:27.061064 12772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:27.061062 12777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 LEADER]: Becoming Leader. State: Replica: 9193e24069244623b206eb66bd093c34, State: Running, Role: LEADER
I20260929 03:27:27.061335 12777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } }
I20260929 03:27:27.061699 12778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } }
I20260929 03:27:27.061794 12778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:27.061949 12779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9193e24069244623b206eb66bd093c34. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } }
I20260929 03:27:27.061995 12779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:27.062418 12786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:27.063110 12786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:27.064807 12786 catalog_manager.cc:1347] Generated new cluster ID: d44b12c1a1c34da7b48b6b58c84eda7b
I20260929 03:27:27.064862 12786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:27.068619 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.058078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.6.116.190:46193)
0929 03:27:27.058269 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.058276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.058296 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:27:27.059170 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.059177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.059226 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.059369 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.060166 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.061271 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.062384 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.062968 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.063104 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.065591 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.065595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.065595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.065606 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.067163 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.067497 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.067500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.067500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.067608 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.067804 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.067805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.067805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.068050 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.068212 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.068280 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.068414 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260929 03:27:27.094549 12786 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:27.095080 12786 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:27.103763 12786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Generated new TSK 0
I20260929 03:27:27.103968 12786 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
I20260929 03:27:27.138113 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.230058 12798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.230232 12798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.230253 12798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.231364 12798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.231400 12798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.231441 12798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:27.232842 12798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:27.233055 12798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.233248 12798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.237715 12804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.237761 12798 server_base.cc:1034] running on GCE node
W20260929 03:27:27.237768 12803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.237725 12806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.238090 12798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.238276 12798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.239426 12798 hybrid_clock.cc:648] HybridClock initialized: now 1790652447239414 us; error 28 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.241844 12798 init.cc:377] Logged in from keytab as oryx/127.6.116.129@KRBTEST.COM (short username oryx)
I20260929 03:27:27.242236 12798 webserver.cc:460] Webserver started at http://127.6.116.129:39215/ using document root <none> and password file <none>
I20260929 03:27:27.242435 12798 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.242506 12798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.242636 12798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.243619 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "c203b981a1e44403820f11e26e54ca4f"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "89f8cc867624328ebe2a9858b8318b90"
server_key_iv: "8c354adf31543e70fb3a065bc0fe7b2b"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.243943 12798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "c203b981a1e44403820f11e26e54ca4f"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "89f8cc867624328ebe2a9858b8318b90"
server_key_iv: "8c354adf31543e70fb3a065bc0fe7b2b"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.245177 12798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:27.245874 12813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.246066 12798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:27.246172 12798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "c203b981a1e44403820f11e26e54ca4f"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "89f8cc867624328ebe2a9858b8318b90"
server_key_iv: "8c354adf31543e70fb3a065bc0fe7b2b"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.246264 12798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.268402 12798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.270725 12798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.270900 12798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.271135 12798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:27.271430 12798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:27.271483 12798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.271531 12798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:27.271561 12798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.277212 12798 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:41905
I20260929 03:27:27.277244 12926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:41905 every 8 connection(s)
I20260929 03:27:27.277559 12798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:27.282802 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12798
I20260929 03:27:27.282899 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:27.283103 6610 external_mini_cluster.cc:1428] Setting key a3d2e6ac5c0e18a49400b272921ba1ba
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:27.287557 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.278221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:56801 (local address 127.6.116.190:46193)
0929 03:27:27.278280 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.278282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.278504 (+ 222us) server_negotiation.cc:407] Connection header received
0929 03:27:27.278941 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.278942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.278951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.279001 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.280250 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.280852 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.281912 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.282113 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.282169 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.284719 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.284720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.284721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.284728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.286100 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.286545 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.286546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.286547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.286604 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.286854 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.286855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.286856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.287007 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.287114 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.287370 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.287457 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:27:27.287703 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.278184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:56801)
0929 03:27:27.278365 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.278404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.278868 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.279079 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.279082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.279816 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.280081 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.280088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.280927 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.280928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.281839 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.281842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.282209 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.282210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.282356 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.282871 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.282881 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.284635 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.286225 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.286231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.286233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.286446 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.286671 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.286671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.286672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.286780 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.287107 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.287113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.287244 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.287348 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.287489 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260929 03:27:27.288117 12927 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:27.288197 12927 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:27.288354 12927 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:27.288810 12736 ts_manager.cc:194] Registered new tserver with Master: c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:27.289368 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:56801
WARNING: no policy specified for oryx/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
I20260929 03:27:27.307019 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.383394 12932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.383597 12932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.383632 12932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.384711 12932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.384769 12932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.384810 12932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:27.386201 12932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:27.386444 12932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.386662 12932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.391321 12938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.391309 12937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.391319 12940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.391501 12932 server_base.cc:1034] running on GCE node
I20260929 03:27:27.391688 12932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.391937 12932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.393100 12932 hybrid_clock.cc:648] HybridClock initialized: now 1790652447393088 us; error 34 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.395494 12932 init.cc:377] Logged in from keytab as oryx/127.6.116.130@KRBTEST.COM (short username oryx)
I20260929 03:27:27.395906 12932 webserver.cc:460] Webserver started at http://127.6.116.130:43621/ using document root <none> and password file <none>
I20260929 03:27:27.396118 12932 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.396194 12932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.396347 12932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.397281 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "ebcbc50e1e9444c1ac56025cbb390024"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "9c320a968010dfd17ae10158a11435f9"
server_key_iv: "a2adde79591a2be8700a92c501a58589"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.397653 12932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "ebcbc50e1e9444c1ac56025cbb390024"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "9c320a968010dfd17ae10158a11435f9"
server_key_iv: "a2adde79591a2be8700a92c501a58589"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.398945 12932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:27.399700 12947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.399861 12932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:27.399977 12932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "ebcbc50e1e9444c1ac56025cbb390024"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "9c320a968010dfd17ae10158a11435f9"
server_key_iv: "a2adde79591a2be8700a92c501a58589"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.400055 12932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.418707 12932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.420436 12932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.420593 12932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.420805 12932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:27.421072 12932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:27.421120 12932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.421154 12932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:27.421180 12932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.426695 12932 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:33923
I20260929 03:27:27.426728 13060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:33923 every 8 connection(s)
I20260929 03:27:27.427012 12932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:27.431176 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 12932
I20260929 03:27:27.431280 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:27.431450 6610 external_mini_cluster.cc:1428] Setting key b61820bcaa3af5fb50cb2b728b3e1fd3
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:27.435552 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.427638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:35313 (local address 127.6.116.190:46193)
0929 03:27:27.427680 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.427682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.427833 (+ 151us) server_negotiation.cc:407] Connection header received
0929 03:27:27.428221 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.428222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.428231 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.428277 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.429035 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.429508 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.430355 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.430543 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.430590 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.432703 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.432704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.432705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.432712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.434117 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.434594 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.434595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.434595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.434642 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.434892 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.434893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.434893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.434991 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.435095 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.435409 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.435493 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:27.435801 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.427623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:35313)
0929 03:27:27.427747 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.427776 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.428149 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.428321 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.428324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.428803 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.428963 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.428967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.429577 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.429578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.430290 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.430292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.430607 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.430607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.430694 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.431107 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.431117 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.432617 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.434202 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.434207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.434209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.434513 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.434704 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.434705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.434706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.434833 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.435107 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.435112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.435264 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.435376 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.435556 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260929 03:27:27.436287 13061 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:27.436365 13061 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:27.436568 13061 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:27.436975 12736 ts_manager.cc:194] Registered new tserver with Master: ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923)
I20260929 03:27:27.437294 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:35313
WARNING: no policy specified for oryx/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
I20260929 03:27:27.457197 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.531014 13066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.531196 13066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.531219 13066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.532202 13066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.532231 13066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.532254 13066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:27.533510 13066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:27.533679 13066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.533828 13066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.538476 13071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.538477 13074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.538491 13066 server_base.cc:1034] running on GCE node
W20260929 03:27:27.538511 13072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.538983 13066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.539242 13066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.540391 13066 hybrid_clock.cc:648] HybridClock initialized: now 1790652447540388 us; error 29 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.542554 13066 init.cc:377] Logged in from keytab as oryx/127.6.116.131@KRBTEST.COM (short username oryx)
I20260929 03:27:27.542922 13066 webserver.cc:460] Webserver started at http://127.6.116.131:35255/ using document root <none> and password file <none>
I20260929 03:27:27.543169 13066 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.543212 13066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.543344 13066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.544245 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "3d69323866924d329b10d2a58cc03d8e"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "952c6925c3a1066c70c139737af6d924"
server_key_iv: "f3322646647f88eb338d2aea76cbfc68"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.544561 13066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "3d69323866924d329b10d2a58cc03d8e"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "952c6925c3a1066c70c139737af6d924"
server_key_iv: "f3322646647f88eb338d2aea76cbfc68"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.545784 13066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:27.546422 13081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.546576 13066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:27.546665 13066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "3d69323866924d329b10d2a58cc03d8e"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "952c6925c3a1066c70c139737af6d924"
server_key_iv: "f3322646647f88eb338d2aea76cbfc68"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.546741 13066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.556465 13066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.558526 13066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.558672 13066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.558889 13066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:27.559228 13066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:27.559273 13066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.559309 13066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:27.559332 13066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.564859 13066 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:46123
I20260929 03:27:27.564891 13194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:46123 every 8 connection(s)
I20260929 03:27:27.565198 13066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:27.570981 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13066
I20260929 03:27:27.571117 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:27.571303 6610 external_mini_cluster.cc:1428] Setting key bf06430fe98b2c465aeb135950dcf30e
I20260929 03:27:27.573632 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.565809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:58329 (local address 127.6.116.190:46193)
0929 03:27:27.565867 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.565869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.566066 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:27:27.566578 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.566580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.566592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.566650 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.567544 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.568029 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.568956 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.569161 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.569238 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.571248 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.571249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.571250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.571256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.572644 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.572968 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.572968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.572968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.573014 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.573176 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.573176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.573176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.573268 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.573314 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.573522 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.573574 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:27:27.573791 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.565804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:58329)
0929 03:27:27.565959 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.565989 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.566475 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.566713 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.566716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.567282 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.567448 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.567452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.568111 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.568112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.568861 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.568863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.569248 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.569249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.569355 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.569823 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.569831 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.571159 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.572707 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.572712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.572714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.572907 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.573053 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.573053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.573053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.573132 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.573317 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.573321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.573414 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.573492 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.573612 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260929 03:27:27.574299 13195 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:27.574388 13195 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:27.574582 13195 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:27.574965 12736 ts_manager.cc:194] Registered new tserver with Master: 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:27.575350 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:58329
I20260929 03:27:27.582919 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.189.keytab.
Entry for principal oryx/127.6.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.189.keytab.
I20260929 03:27:27.606093 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:43203
--webserver_interface=127.6.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.189.keytab
--principal=oryx/127.6.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:46193,127.6.116.189:43203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.683341 13200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.683516 13200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.683537 13200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.684602 13200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:27.684645 13200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.684659 13200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.684671 13200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:27.684713 13200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:27.686086 13200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:46193,127.6.116.189:43203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.189.keytab
--principal=oryx/127.6.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:43203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
W20260929 03:27:27.686221 13200 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.116.190:46193,127.6.116.189:43203'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 03:27:27.686352 13200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.686556 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.690814 13205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.690894 13206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.690934 13208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.690979 13200 server_base.cc:1034] running on GCE node
I20260929 03:27:27.691264 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.691519 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.692651 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790652447692636 us; error 29 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.694875 13200 init.cc:377] Logged in from keytab as oryx/127.6.116.189@KRBTEST.COM (short username oryx)
I20260929 03:27:27.695312 13200 webserver.cc:460] Webserver started at http://127.6.116.189:39955/ using document root <none> and password file <none>
I20260929 03:27:27.695487 13200 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.695525 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.695613 13200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.696491 13200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/instance:
uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "b6c0d23ae88eb1ff41df4691fe227bf1"
server_key_iv: "370cb4fc3df863b2117a5f6aae16f596"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.696763 13200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal/instance:
uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "b6c0d23ae88eb1ff41df4691fe227bf1"
server_key_iv: "370cb4fc3df863b2117a5f6aae16f596"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.697837 13200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:27.698400 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.698559 13200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:27.698683 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "b6c0d23ae88eb1ff41df4691fe227bf1"
server_key_iv: "370cb4fc3df863b2117a5f6aae16f596"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.698753 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.713281 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.714756 13200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.714892 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.718302 13200 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.189:43203
I20260929 03:27:27.718331 13267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.189:43203 every 8 connection(s)
I20260929 03:27:27.718621 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
I20260929 03:27:27.719485 13268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:27.719971 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13200
I20260929 03:27:27.721138 13268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.189@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
I20260929 03:27:27.729529 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.721755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.6.116.190:46193)
0929 03:27:27.721803 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.721805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.721921 (+ 116us) server_negotiation.cc:407] Connection header received
0929 03:27:27.721997 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.721998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.722008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.722052 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.722487 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.723011 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.723927 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.724133 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.724212 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.726324 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.726325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.726325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.726332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.727975 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.728499 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.728500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.728501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.728577 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.728856 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.728857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.728858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.729018 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.729098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.729355 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.729436 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:27:27.729755 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.721687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42408)
0929 03:27:27.721831 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.721866 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.721937 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.722073 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.722076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.722318 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.722398 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.722401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.723149 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.723150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.723837 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.723839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.724221 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.724222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.724300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.724756 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.724763 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.726221 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.728067 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.728071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.728073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.728403 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.728678 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.728679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.728679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.728789 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.729090 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.729095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.729229 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.729334 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.729483 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260929 03:27:27.729760 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.720508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.6.116.189:43203)
0929 03:27:27.720674 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.720681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.720711 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:27:27.721177 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.721184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.721239 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.721376 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.722071 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.722848 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.723656 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.723971 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.724080 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.726782 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.726789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.726789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.726796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.728571 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.728911 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.728914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.728914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.728969 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.729168 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.729169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.729169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.729300 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.729445 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.729541 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.729617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260929 03:27:27.730458 13268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.189@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
I20260929 03:27:27.736282 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.730710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54478)
0929 03:27:27.730751 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.730763 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.730802 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.730928 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.730929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.731033 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.731126 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.731128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.731712 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.731713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.732189 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.732191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.732511 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.732512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.732569 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.732993 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.732997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.734082 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.735538 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.735539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.735539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.735685 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.735846 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.735846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.735846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.735892 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.736051 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.736051 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.736080 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.736128 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.736207 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:27:27.736375 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.730763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.6.116.189:43203)
0929 03:27:27.730797 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.730799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.730809 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:27.730851 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.730852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.730860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.730924 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.731181 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.731623 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.732260 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.732437 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.732486 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.734156 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.734157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.734157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.734162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.735472 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.735748 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.735749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.735749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.735793 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.735929 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.735929 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.735929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.736007 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.736049 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.736141 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.736248 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":17}
I20260929 03:27:27.737573 13268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Bootstrap starting.
I20260929 03:27:27.738308 13268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:27.738528 13268 log.cc:824] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:27.739292 13268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: No bootstrap required, opened a new log
I20260929 03:27:27.740603 13268 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } }
I20260929 03:27:27.740723 13268 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:27.740746 13268 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e8c5b4b0a542a19fdab6f324188b74, State: Initialized, Role: FOLLOWER
I20260929 03:27:27.740826 13268 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } }
I20260929 03:27:27.741045 13268 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:27.741047 13274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } } }
I20260929 03:27:27.741181 13274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:27:27.742499 13285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:27.742554 13285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:27.744050 13200 master_runner.cc:428] Detected that this master b4e8c5b4b0a542a19fdab6f324188b74 is joining an existing cluster
I20260929 03:27:27.744100 13200 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.116.189:43203
I20260929 03:27:27.751814 13291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.744989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54494)
0929 03:27:27.745211 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.745219 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.745281 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.745422 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.745423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.745634 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.745700 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.745701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.746334 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.746335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.746871 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.746872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.747266 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.747267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.747322 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.747733 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.747738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.748119 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.750211 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.750212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.750213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.750426 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.750630 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.750630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.750631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.750733 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.751427 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.751427 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.751460 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.751537 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.751687 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260929 03:27:27.751821 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.745152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.6.116.189:43203)
0929 03:27:27.745186 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.745187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.745262 (+ 75us) server_negotiation.cc:407] Connection header received
0929 03:27:27.745347 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.745348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.745355 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.745420 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.745770 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.746253 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.746956 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.747178 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.747255 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.748234 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.748235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.748236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.748242 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.750107 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.750503 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.750505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.750505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.750564 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.750799 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.750800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.750800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.750920 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.750964 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.751559 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.751739 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":18}
I20260929 03:27:27.752717 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.745018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.6.116.190:46193)
0929 03:27:27.745065 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.745067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.745233 (+ 166us) server_negotiation.cc:407] Connection header received
0929 03:27:27.745311 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.745312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.745322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.745380 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.745919 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.746685 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.747746 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.748041 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.748119 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.749234 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.749235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.749236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.749243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.751337 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.751777 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.751778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.751779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.751836 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.752184 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.752185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.752186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.752310 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.752353 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.752555 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.752635 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:27:27.752938 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.744979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42416)
0929 03:27:27.745171 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.745182 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.745247 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.745434 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.745435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.745741 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.745813 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.745816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.746784 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.746784 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.747668 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.747671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.748127 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.748128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.748196 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.748584 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.748589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.749166 (+ 577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.751406 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.751406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.751407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.751683 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.752042 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.752043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.752043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.752103 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.752399 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.752399 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.752447 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.752515 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.752607 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":145,"threads_started":1}
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.190@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
I20260929 03:27:27.761375 13271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.754964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.6.116.189:43203)
0929 03:27:27.755002 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.755004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.755150 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:27:27.755246 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.755247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.755257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.755312 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.755719 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.756133 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.756881 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.757072 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.757149 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.758980 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.758981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.758982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.758988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.760262 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.760644 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.760645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.760645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.760701 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.760890 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.760891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.760891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.760992 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.761062 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.761227 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.761295 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:27:27.761497 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.754891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54500)
0929 03:27:27.755017 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.755087 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.755176 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.755318 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.755322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.755579 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.755620 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.755622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.756213 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.756214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.756786 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.756788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.757152 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.757153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.757231 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.757662 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.757666 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.758882 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.760355 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.760363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.760365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.760557 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.760766 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.760767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.760767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.760825 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.761055 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.761062 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.761111 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.761204 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.761332 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 03:27:27.761996 12736 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.116.189:43203: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
member_type: NON_VOTER
last_known_addr {
host: "127.6.116.189"
port: 43203
}
attrs {
promote: true
}
}
dest_uuid: "9193e24069244623b206eb66bd093c34"
cas_config_opid_index: -1
I20260929 03:27:27.762282 12736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } }
I20260929 03:27:27.762773 13294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b4e8c5b4b0a542a19fdab6f324188b74 (127.6.116.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } }
I20260929 03:27:27.762975 13242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:27.763844 13294 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.116.189:43203
I20260929 03:27:27.763850 13242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 03:27:27.764043 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } }
I20260929 03:27:27.764092 13200 master.cc:537] Master@127.6.116.189:43203 shutting down...
I20260929 03:27:27.764292 13294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:27.765021 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
W20260929 03:27:27.765667 12720 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 03:27:27.766230 12720 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 -> Peer b4e8c5b4b0a542a19fdab6f324188b74 (127.6.116.189:43203): Couldn't send request to peer b4e8c5b4b0a542a19fdab6f324188b74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:27:27.766556 13200 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:27.766647 13200 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:27.766703 13200 tablet_replica.cc:330] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: stopping tablet replica
I20260929 03:27:27.778954 13200 master.cc:552] Master@127.6.116.189:43203 shutdown complete.
I20260929 03:27:27.779014 13200 master_runner.cc:305] Clearing existing system tablet
I20260929 03:27:27.779166 13200 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.779232 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.779810 13200 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:27.780251 13299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.780400 13200 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.780457 13200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.780480 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "b6c0d23ae88eb1ff41df4691fe227bf1"
server_key_iv: "370cb4fc3df863b2117a5f6aae16f596"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.781613 13200 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:27.782974 13200 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:27:27.783030 13200 log.cc:1196] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 03:27:27.783306 13200 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Deleting consensus metadata
I20260929 03:27:27.783434 13200 master_runner.cc:315] Copying system tablet from 127.6.116.190:46193
I20260929 03:27:27.784008 13200 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: tablet copy: Beginning tablet copy session from remote peer at address 127.6.116.190:46193
I20260929 03:27:27.789873 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.784279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42424)
0929 03:27:27.784384 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.784394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.784460 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.784592 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.784593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.784771 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.784822 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.784823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.785464 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.785465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.786013 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.786014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.786384 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.786385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.786452 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.786902 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.786906 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.787235 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.788995 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.788996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.788996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.789156 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.789351 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.789351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.789352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.789400 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.789643 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.789643 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.789676 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.789743 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.789811 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260929 03:27:27.789925 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.784335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.6.116.190:46193)
0929 03:27:27.784383 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.784386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.784443 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:27:27.784520 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.784521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.784530 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.784586 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.784901 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.785388 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.786095 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.786316 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.786386 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.787316 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.787317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.787317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.787325 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.788920 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.789223 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.789224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.789225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.789287 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.789458 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.789459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.789459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.789586 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.789661 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.789765 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.789838 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:27:27.790216 12756 tablet_copy_service.cc:140] P 9193e24069244623b206eb66bd093c34: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b4e8c5b4b0a542a19fdab6f324188b74 ({username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:42424)
I20260929 03:27:27.790303 12756 tablet_copy_service.cc:161] P 9193e24069244623b206eb66bd093c34: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b4e8c5b4b0a542a19fdab6f324188b74 at {username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:42424: session id = b4e8c5b4b0a542a19fdab6f324188b74-00000000000000000000000000000000
I20260929 03:27:27.791091 12756 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:27:27.791604 13200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:27:27.792409 13200 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: tablet copy: Starting download of 0 data blocks...
I20260929 03:27:27.792560 13200 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: tablet copy: Starting download of 1 WAL segments...
I20260929 03:27:27.793414 13200 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:27:27.794144 12756 tablet_copy_service.cc:342] P 9193e24069244623b206eb66bd093c34: Request end of tablet copy session b4e8c5b4b0a542a19fdab6f324188b74-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:42424
I20260929 03:27:27.794211 12756 tablet_copy_service.cc:433] P 9193e24069244623b206eb66bd093c34: ending tablet copy session b4e8c5b4b0a542a19fdab6f324188b74-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b4e8c5b4b0a542a19fdab6f324188b74
I20260929 03:27:27.796226 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:27.797256 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.797299 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.799834 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.799969 13200 server_base.cc:1034] running on GCE node
I20260929 03:27:27.800063 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.800179 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.801319 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790652447801295 us; error 39 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.802539 13200 init.cc:377] Logged in from keytab as oryx/127.6.116.189@KRBTEST.COM (short username oryx)
I20260929 03:27:27.802809 13200 webserver.cc:460] Webserver started at http://127.6.116.189:35847/ using document root <none> and password file <none>
I20260929 03:27:27.802947 13200 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.802984 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.803633 13200 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.804112 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.804229 13200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.804275 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "b4e8c5b4b0a542a19fdab6f324188b74"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "b6c0d23ae88eb1ff41df4691fe227bf1"
server_key_iv: "370cb4fc3df863b2117a5f6aae16f596"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.804324 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.813158 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.814730 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.817655 13200 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.189:43203
I20260929 03:27:27.817847 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.189:43203 every 8 connection(s)
I20260929 03:27:27.818014 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
I20260929 03:27:27.818763 13369 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:27:27.819337 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Bootstrap starting.
I20260929 03:27:27.822032 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:27:27.822269 13369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Bootstrap complete.
I20260929 03:27:27.822476 13369 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } }
I20260929 03:27:27.822561 13369 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4e8c5b4b0a542a19fdab6f324188b74, State: Initialized, Role: LEARNER
I20260929 03:27:27.822600 13369 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } }
I20260929 03:27:27.822770 13369 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:27.822773 13372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } }
I20260929 03:27:27.822897 13372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:27:27.824241 13383 catalog_manager.cc:1259] Loaded cluster ID: d44b12c1a1c34da7b48b6b58c84eda7b
I20260929 03:27:27.824285 13383 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: loading cluster ID for follower catalog manager: success
I20260929 03:27:27.825382 13383 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: acquiring CA information for follower catalog manager: success
I20260929 03:27:27.825629 13383 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:27:27.838220 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.831639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.6.116.189:43203)
0929 03:27:27.831737 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.831744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.831751 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:27.831768 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.831768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.831775 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.831821 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.832473 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.833096 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.834154 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.834342 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.834385 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.835752 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.835753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.835754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.835759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.837065 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.837432 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.837432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.837433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.837478 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.837688 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.837689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.837689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.837771 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.837870 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.838033 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.838099 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":43,"threads_started":1}
WARNING: no policy specified for oryx/127.6.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.188.keytab.
Entry for principal oryx/127.6.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.188.keytab.
I20260929 03:27:27.861675 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:41127
--webserver_interface=127.6.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.188.keytab
--principal=oryx/127.6.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:27.940253 13387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:27.940452 13387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:27.940488 13387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:27.941578 13387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:27.941637 13387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:27.941661 13387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:27.941690 13387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:27.941713 13387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:27.943185 13387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.188.keytab
--principal=oryx/127.6.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:27.943462 13387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:27.943665 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:27:27.948140 13387 server_base.cc:1034] running on GCE node
W20260929 03:27:27.948078 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.948079 13392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:27.948107 13393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:27.948510 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:27.948807 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:27.949936 13387 hybrid_clock.cc:648] HybridClock initialized: now 1790652447949925 us; error 26 us; skew 500 ppm
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:27.952194 13387 init.cc:377] Logged in from keytab as oryx/127.6.116.188@KRBTEST.COM (short username oryx)
I20260929 03:27:27.952570 13387 webserver.cc:460] Webserver started at http://127.6.116.188:37675/ using document root <none> and password file <none>
I20260929 03:27:27.952804 13387 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:27.952867 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:27.953002 13387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:27.953926 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/instance:
uuid: "bee18762a91345ba8363fbd9ab49a782"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "531f899e207f75a8b64b272fe78c78e6"
server_key_iv: "31aad7e8cff8308f74385229f20d3828"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.954253 13387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal/instance:
uuid: "bee18762a91345ba8363fbd9ab49a782"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "531f899e207f75a8b64b272fe78c78e6"
server_key_iv: "31aad7e8cff8308f74385229f20d3828"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.955559 13387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:27.956233 13402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:27.956377 13387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:27.956485 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "bee18762a91345ba8363fbd9ab49a782"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "531f899e207f75a8b64b272fe78c78e6"
server_key_iv: "31aad7e8cff8308f74385229f20d3828"
server_key_version: "encryptionkey@0"
I20260929 03:27:27.956554 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:27.970969 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:27.972602 13387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:27.972746 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:27.976311 13387 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.188:41127
I20260929 03:27:27.976346 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.188:41127 every 8 connection(s)
I20260929 03:27:27.976636 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
I20260929 03:27:27.977516 13455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:27.978825 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.188@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:27.986042 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13387
I20260929 03:27:27.986714 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.979542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.6.116.190:46193)
0929 03:27:27.979596 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.979597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.979664 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:27:27.980055 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.980056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.980065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.980124 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.980604 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.981080 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.981966 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.982217 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.982278 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.984336 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.984338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.984338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.984344 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.985447 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.985780 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.985781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.985781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.985829 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.986050 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.986051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.986052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.986193 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.986283 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.986523 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.986612 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:27:27.986874 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.979411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42432)
0929 03:27:27.979556 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.979592 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.979985 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.980161 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.980164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.980373 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.980536 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.980539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.981145 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.981146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.981886 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.981888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.982300 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.982301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.982392 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.982835 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.982842 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.984259 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.985511 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.985516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.985517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.985719 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.985896 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.985896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.985896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.985989 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.986273 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.986278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.986375 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.986483 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.986646 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260929 03:27:27.987378 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.188@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
I20260929 03:27:27.993885 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.987583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54536)
0929 03:27:27.987623 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.987632 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.987667 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.987801 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.987801 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.987923 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.987963 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.987964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.988686 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.988687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.989173 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.989174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.989561 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.989562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.989625 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.990010 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.990014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.991242 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.992959 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.992960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.992960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.993203 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.993373 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.993374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.993374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:27.993434 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:27.993637 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:27.993638 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:27.993680 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:27:27.993755 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:27:27.993829 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:27:27.993968 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.987652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.6.116.189:43203)
0929 03:27:27.987699 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.987701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.987710 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:27.987732 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.987733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.987745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.987794 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.988034 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.988577 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.989260 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.989467 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.989560 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.991331 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.991332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.991332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.991339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.992873 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.993268 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.993269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.993269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.993316 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.993498 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.993499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.993499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.993583 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.993639 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.993783 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.993859 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:27.994407 13455 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:27:27.995007 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.986516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.6.116.188:41127)
0929 03:27:27.986660 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.986664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.986679 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:27:27.986710 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.986712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.986724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.987551 (+ 827us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.987592 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.988266 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.988967 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.989260 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.989355 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.991906 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.991913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.991913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.991920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.993701 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.994053 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.994056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.994056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.994114 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:27.994430 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:27.994430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:27.994431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:27.994601 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:27.994796 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:27.994862 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:27:27.994918 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
Sep 29 03:27:27 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.188@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
I20260929 03:27:28.000627 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.994626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50252)
0929 03:27:27.994667 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:27.994676 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:27.994709 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:27.995210 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:27.995210 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:27.995320 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:27.995340 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.995341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.995972 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.995973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.996418 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:27.996419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.996730 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.996731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:27.996766 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.997112 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:27.997116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:27.998110 (+ 994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:27.999919 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:27.999920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:27.999920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.000069 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.000225 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.000225 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.000225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.000283 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.000434 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.000435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.000463 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.000510 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.000571 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:27:28.000726 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:27.994677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50252 (local address 127.6.116.188:41127)
0929 03:27:27.995024 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:27.995026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:27.995031 (+ 5us) server_negotiation.cc:407] Connection header received
0929 03:27:27.995089 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:27.995090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:27.995100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:27.995175 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:27.995405 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.995908 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.996479 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:27.996677 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:27.996724 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:27.998182 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:27.998182 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:27.998183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:27.998188 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:27.999858 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.000126 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.000126 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.000127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.000171 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.000318 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.000319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.000319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.000392 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.000433 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.000522 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.000611 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":332,"thread_start_us":64,"threads_started":1}
I20260929 03:27:28.001684 13455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Bootstrap starting.
I20260929 03:27:28.002202 13455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:28.002527 13455 log.cc:824] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:28.003296 13455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: No bootstrap required, opened a new log
I20260929 03:27:28.004676 13455 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } }
I20260929 03:27:28.004776 13455 raft_consensus.cc:374] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:28.004825 13455 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bee18762a91345ba8363fbd9ab49a782, State: Initialized, Role: FOLLOWER
I20260929 03:27:28.004937 13455 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } }
I20260929 03:27:28.005180 13461 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } } }
I20260929 03:27:28.005247 13455 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:28.005264 13461 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:27:28.006692 13472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:27:28.006747 13472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:27:28.008699 13387 master_runner.cc:428] Detected that this master bee18762a91345ba8363fbd9ab49a782 is joining an existing cluster
I20260929 03:27:28.008750 13387 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.116.188:41127
I20260929 03:27:28.015946 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.009685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54546)
0929 03:27:28.009805 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.009815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.009857 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.010221 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.010222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.010427 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.010476 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.010477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011331 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.011332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012162 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.012164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012656 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012735 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.013146 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.013151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.013426 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.014980 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.014981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.014981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.015238 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.015446 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.015446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.015447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.015498 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.015735 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.015736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.015774 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.015818 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.015883 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260929 03:27:28.016001 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.010035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.6.116.189:43203)
0929 03:27:28.010102 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.010104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.010112 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:28.010131 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.010131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.010139 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.010211 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.010553 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011251 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012258 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012532 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012569 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.013507 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.013508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.013508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.013515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.014908 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.015314 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.015315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.015315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.015373 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.015552 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.015552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.015553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.015672 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.015746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.015843 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.015916 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:27:28.042945 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.009868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50258)
0929 03:27:28.009968 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.009976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.010031 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.010371 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.010371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.010567 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.010619 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.010621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011490 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.011491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012280 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.012282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012760 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012822 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.039676 (+ 26854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.039680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.040070 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.041856 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.041857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.041857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.042098 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.042309 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042309 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.042310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.042360 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.042613 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.042613 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.042646 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.042710 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.042810 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260929 03:27:28.043016 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.010069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.6.116.188:41127)
0929 03:27:28.010252 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.010253 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.010260 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:28.010276 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.010277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.010285 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.010341 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.010698 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011407 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012368 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012642 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012678 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.040134 (+ 27456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.040135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.040135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.040141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.041766 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042169 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.042170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.042171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.042235 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042419 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.042420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.042420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.042537 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.042575 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.042747 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.042882 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":165}
I20260929 03:27:28.043493 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.009671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.6.116.190:46193)
0929 03:27:28.009715 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.009717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.009848 (+ 131us) server_negotiation.cc:407] Connection header received
0929 03:27:28.009938 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.009939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.009950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.009980 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.010449 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011197 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012259 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012533 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012569 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.040099 (+ 27530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.040100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.040101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.040108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.041885 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042289 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.042290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.042291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.042358 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042559 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.042560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.042560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.042678 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.042755 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.042930 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.043006 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:28.043747 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.009625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42448)
0929 03:27:28.009786 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.009797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.009868 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.010059 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.010060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.010285 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.010359 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.010361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.011286 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.011287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012162 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.012164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.012653 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.012654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.012735 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.039507 (+ 26772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.039512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.040002 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.041960 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.041961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.041961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.042214 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.042418 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.042419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.042419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.042496 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.042781 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.042781 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.042817 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.043593 (+ 776us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.043688 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":47,"thread_start_us":86,"threads_started":1}
I20260929 03:27:28.046029 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12798
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652447, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.190@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
I20260929 03:27:28.053371 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.045657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50266)
0929 03:27:28.045699 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.045709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.045745 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.046015 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.046016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.046172 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.046233 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.046235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.047109 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.047110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.048053 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.048055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.048518 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.048519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.048590 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.048996 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.049001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.050668 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.052407 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.052408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.052408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.052605 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.052794 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.052795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.052795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.052847 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.053085 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.053086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.053124 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.053182 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.053281 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:27:28.053606 13459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.045702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.6.116.188:41127)
0929 03:27:28.045735 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.045737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.045745 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:28.045905 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.045905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.045913 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.045969 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.046316 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.046975 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.048136 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.048442 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.048512 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.050751 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.050752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.050753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.050759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.052320 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.052677 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.052678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.052679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.052729 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.052906 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.052907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.052907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.053011 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.053062 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.053467 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.053531 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:28.054502 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:41905
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:46193
--webserver_port=39215
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:27:28.055454 12736 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.116.188:41127: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bee18762a91345ba8363fbd9ab49a782"
member_type: NON_VOTER
last_known_addr {
host: "127.6.116.188"
port: 41127
}
attrs {
promote: true
}
}
dest_uuid: "9193e24069244623b206eb66bd093c34"
cas_config_opid_index: 5
I20260929 03:27:28.055658 12736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } }
I20260929 03:27:28.055999 13295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bee18762a91345ba8363fbd9ab49a782 (127.6.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.056372 13429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:28.057024 13295 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.116.188:41127
I20260929 03:27:28.057096 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.057175 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.057250 13429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:27:28.057499 13387 master.cc:537] Master@127.6.116.188:41127 shutting down...
I20260929 03:27:28.057490 13294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:27:28.058171 12720 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 -> Peer bee18762a91345ba8363fbd9ab49a782 (127.6.116.188:41127): Couldn't send request to peer bee18762a91345ba8363fbd9ab49a782. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 03:27:28.062258 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.056057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54552)
0929 03:27:28.058278 (+ 2221us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.058287 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.058326 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.058513 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.058514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.058515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.058546 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.058547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.059544 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.059545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.061168 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.061171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.061967 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.061968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.062043 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.062065 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.062128 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.062170 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2209}
I20260929 03:27:28.062384 13387 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:28.062467 13387 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:28.062517 13387 tablet_replica.cc:330] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: stopping tablet replica
I20260929 03:27:28.062803 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.056136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.6.116.189:43203)
0929 03:27:28.058252 (+ 2116us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.058254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.058373 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:27:28.058395 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.058395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.058416 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.058467 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.058610 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.059417 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.061291 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.061898 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.061963 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.062027 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.062651 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.062717 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2095}
I20260929 03:27:28.063165 13343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 LEARNER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 03:27:28.063370 13294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 03:27:28.064079 13343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bee18762a91345ba8363fbd9ab49a782 (127.6.116.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.065272 13372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9193e24069244623b206eb66bd093c34. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.065374 13372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:27:28.065416 13372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: NON_VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: true } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.065478 13372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:27:28.066171 13294 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: attempting to promote NON_VOTER b4e8c5b4b0a542a19fdab6f324188b74 to VOTER
I20260929 03:27:28.066385 13294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } }
I20260929 03:27:28.066640 13387 master.cc:552] Master@127.6.116.188:41127 shutdown complete.
I20260929 03:27:28.066674 13387 master_runner.cc:305] Clearing existing system tablet
I20260929 03:27:28.066800 13387 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:28.066836 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:28.066960 13343 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 LEARNER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 03:27:28.067144 13295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 03:27:28.067492 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.066837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50274)
0929 03:27:28.067429 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.067469 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578}
I20260929 03:27:28.067700 13387 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:27:28.067744 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.067683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50282)
0929 03:27:28.067709 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.067727 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:41127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 03:27:28.067793 12720 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 -> Peer bee18762a91345ba8363fbd9ab49a782 (127.6.116.188:41127): Couldn't send request to peer bee18762a91345ba8363fbd9ab49a782. Status: Network error: Client connection negotiation failed: client connection to 127.6.116.188:41127: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:27:28.068074 13295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b4e8c5b4b0a542a19fdab6f324188b74 (127.6.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.068367 13486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.068491 13387 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:28.068534 13387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:27:28.068557 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "bee18762a91345ba8363fbd9ab49a782"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "531f899e207f75a8b64b272fe78c78e6"
server_key_iv: "31aad7e8cff8308f74385229f20d3828"
server_key_version: "encryptionkey@0"
I20260929 03:27:28.068845 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.068908 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.068950 13295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.068987 13295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.069209 13387 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:28.069213 13343 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b4e8c5b4b0a542a19fdab6f324188b74 (127.6.116.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.070065 13483 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.070143 13483 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:27:28.070425 13387 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 03:27:28.070479 13387 log.cc:1196] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 03:27:28.070654 13387 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Deleting consensus metadata
I20260929 03:27:28.070848 13387 master_runner.cc:315] Copying system tablet from 127.6.116.190:46193
I20260929 03:27:28.071431 13387 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: tablet copy: Beginning tablet copy session from remote peer at address 127.6.116.190:46193
I20260929 03:27:28.077999 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.071658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42450)
0929 03:27:28.071760 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.071769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.071819 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.071941 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.071941 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.072109 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.072150 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.072151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.073029 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.073030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.073836 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.073838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.074267 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.074268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.074329 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.074769 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.074774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.075161 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.076957 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.076958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.076959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.077169 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.077382 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.077383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.077383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.077450 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.077708 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.077709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.077751 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.077823 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.077915 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260929 03:27:28.078011 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.071705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.6.116.190:46193)
0929 03:27:28.071739 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.071740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.071807 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:27:28.071863 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.071864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.071872 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.071935 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.072214 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.072947 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.073922 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.074185 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.074254 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.075265 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.075266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.075266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.075272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.076870 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.077258 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.077259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.077259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.077315 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.077516 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.077517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.077517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.077642 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.077706 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.077856 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.077928 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:28.078536 12756 tablet_copy_service.cc:140] P 9193e24069244623b206eb66bd093c34: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bee18762a91345ba8363fbd9ab49a782 ({username='oryx', principal='oryx/127.6.116.188@KRBTEST.COM'} at 127.0.0.1:42450)
I20260929 03:27:28.078593 12756 tablet_copy_service.cc:161] P 9193e24069244623b206eb66bd093c34: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bee18762a91345ba8363fbd9ab49a782 at {username='oryx', principal='oryx/127.6.116.188@KRBTEST.COM'} at 127.0.0.1:42450: session id = bee18762a91345ba8363fbd9ab49a782-00000000000000000000000000000000
I20260929 03:27:28.079133 12756 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 03:27:28.079598 13387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 03:27:28.080761 13387 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: tablet copy: Starting download of 0 data blocks...
I20260929 03:27:28.080974 13387 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: tablet copy: Starting download of 1 WAL segments...
I20260929 03:27:28.081733 13387 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 03:27:28.082590 12756 tablet_copy_service.cc:342] P 9193e24069244623b206eb66bd093c34: Request end of tablet copy session bee18762a91345ba8363fbd9ab49a782-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.116.188@KRBTEST.COM'} at 127.0.0.1:42450
I20260929 03:27:28.082641 12756 tablet_copy_service.cc:433] P 9193e24069244623b206eb66bd093c34: ending tablet copy session bee18762a91345ba8363fbd9ab49a782-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bee18762a91345ba8363fbd9ab49a782
I20260929 03:27:28.085101 13387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:28.086165 13493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.087791 13494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:28.088127 13387 server_base.cc:1034] running on GCE node
W20260929 03:27:28.088269 13496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:28.088483 13387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:28.088608 13387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:28.089737 13387 hybrid_clock.cc:648] HybridClock initialized: now 1790652448089728 us; error 22 us; skew 500 ppm
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:28.091300 13387 init.cc:377] Logged in from keytab as oryx/127.6.116.188@KRBTEST.COM (short username oryx)
I20260929 03:27:28.091517 13387 webserver.cc:460] Webserver started at http://127.6.116.188:40155/ using document root <none> and password file <none>
I20260929 03:27:28.091634 13387 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:28.091667 13387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:28.092343 13387 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:28.092893 13503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.093053 13387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.093102 13387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "bee18762a91345ba8363fbd9ab49a782"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "531f899e207f75a8b64b272fe78c78e6"
server_key_iv: "31aad7e8cff8308f74385229f20d3828"
server_key_version: "encryptionkey@0"
I20260929 03:27:28.093148 13387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:28.110893 13387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:28.112646 13387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:28.115513 13387 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.188:41127
I20260929 03:27:28.115913 13555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.188:41127 every 8 connection(s)
I20260929 03:27:28.115911 13387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
I20260929 03:27:28.116854 13556 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:27:28.117373 13556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Bootstrap starting.
I20260929 03:27:28.120970 13556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:27:28.121207 13556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: Bootstrap complete.
I20260929 03:27:28.121417 13556 raft_consensus.cc:348] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } }
I20260929 03:27:28.121505 13556 raft_consensus.cc:729] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bee18762a91345ba8363fbd9ab49a782, State: Initialized, Role: LEARNER
I20260929 03:27:28.121541 13556 consensus_queue.cc:260] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } }
I20260929 03:27:28.121737 13556 sys_catalog.cc:564] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:28.121727 13559 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.121932 13559 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: LEARNER
I20260929 03:27:28.123476 13570 catalog_manager.cc:1259] Loaded cluster ID: d44b12c1a1c34da7b48b6b58c84eda7b
I20260929 03:27:28.124006 13570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: loading cluster ID for follower catalog manager: success
I20260929 03:27:28.125149 13570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: acquiring CA information for follower catalog manager: success
I20260929 03:27:28.125414 13570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:27:28.152166 13481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:28.152336 13481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:28.152359 13481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:28.153439 13481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:28.153477 13481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:28.153502 13481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:28.154881 13481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:41905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=39215
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:28.155143 13481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:28.155368 13481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:27:28.160006 13481 server_base.cc:1034] running on GCE node
W20260929 03:27:28.159987 13576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.159981 13575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.160020 13578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:28.160386 13481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:28.160617 13481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:28.161768 13481 hybrid_clock.cc:648] HybridClock initialized: now 1790652448161755 us; error 23 us; skew 500 ppm
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:28.164192 13481 init.cc:377] Logged in from keytab as oryx/127.6.116.129@KRBTEST.COM (short username oryx)
I20260929 03:27:28.164613 13481 webserver.cc:460] Webserver started at http://127.6.116.129:39215/ using document root <none> and password file <none>
I20260929 03:27:28.164824 13481 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:28.164897 13481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:28.166124 13481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:28.166659 13585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.166805 13481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:28.166891 13481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "c203b981a1e44403820f11e26e54ca4f"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "89f8cc867624328ebe2a9858b8318b90"
server_key_iv: "8c354adf31543e70fb3a065bc0fe7b2b"
server_key_version: "encryptionkey@0"
I20260929 03:27:28.167212 13481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:28.182255 13481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:28.184270 13481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:28.184414 13481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:28.184615 13481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:28.184873 13481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:28.184919 13481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.184957 13481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:28.184988 13481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.190555 13481 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:41905
I20260929 03:27:28.190604 13698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:41905 every 8 connection(s)
I20260929 03:27:28.190922 13481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
I20260929 03:27:28.200176 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13481
I20260929 03:27:28.200340 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12932
I20260929 03:27:28.202521 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.192002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:44635 (local address 127.6.116.189:43203)
0929 03:27:28.192054 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.192056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.192201 (+ 145us) server_negotiation.cc:407] Connection header received
0929 03:27:28.192589 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.192590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.192602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.192652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.193443 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194236 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195408 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195685 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195752 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.199137 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.199138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.199138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.199146 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.200786 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201485 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201542 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201821 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201951 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.201991 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.202355 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.202432 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:27:28.202531 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.191628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:54093 (local address 127.6.116.190:46193)
0929 03:27:28.191674 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.191676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.191915 (+ 239us) server_negotiation.cc:407] Connection header received
0929 03:27:28.192507 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.192509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.192522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.192577 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.193471 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194205 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195547 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195787 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195850 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.198575 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.198577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.198577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.198584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.200416 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201485 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201551 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201821 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201941 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.201985 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.202355 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.202433 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:28.202982 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.191903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.6.116.129:44635)
0929 03:27:28.192125 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.192134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.192398 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.192659 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.192660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.193147 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.193353 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.193358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194315 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.194316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195314 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.195316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195764 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195897 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.196412 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.196422 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.199031 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.201083 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.201085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201396 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.201619 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201619 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.201620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201743 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.202044 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.202044 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.202185 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.202636 (+ 451us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.202794 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":156,"thread_start_us":51,"threads_started":1}
I20260929 03:27:28.202981 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.191637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:54093)
0929 03:27:28.191794 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.191837 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.192385 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.192646 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.192649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.193095 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.193353 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.193358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194315 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.194316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195442 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.195444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195901 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195954 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.196430 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.196433 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.198464 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.200996 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.200996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.200997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201396 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.201620 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.201622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201743 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.202033 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.202034 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.202185 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.202636 (+ 451us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.202745 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":95,"spinlock_wait_cycles":6528,"thread_start_us":90,"threads_started":1}
I20260929 03:27:28.203449 13701 heartbeater.cc:347] Connected to a master server at 127.6.116.189:43203
I20260929 03:27:28.203558 13701 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.203646 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.191867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:57061 (local address 127.6.116.188:41127)
0929 03:27:28.192066 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.192068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.192079 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:27:28.192705 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.192706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.192720 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.192826 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.193456 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194235 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195517 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195828 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195902 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.198134 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.198136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.198136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.198142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.200123 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201483 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201545 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201813 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.201814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.201814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.201947 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.202003 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.202940 (+ 937us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.203029 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":47,"threads_started":1}
I20260929 03:27:28.203668 13700 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:28.203701 13700 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.203745 13701 heartbeater.cc:511] Master 127.6.116.189:43203 requested a full tablet report, sending...
I20260929 03:27:28.203747 13700 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:28.204111 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.191637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.6.116.129:57061)
0929 03:27:28.191794 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.191838 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.192385 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.192831 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.192832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.193156 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.193353 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.193358 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.194330 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.194330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195437 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.195440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.195900 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.195901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.195969 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.196594 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.196599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.198049 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.200857 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.200863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.200864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201398 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.201612 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.201613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.201613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.201747 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.202027 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.202031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.202186 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.202304 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.202471 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":153,"thread_start_us":88,"threads_started":1}
I20260929 03:27:28.204206 13333 ts_manager.cc:194] Registered new tserver with Master: c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:28.204553 12736 ts_manager.cc:194] Re-registered known tserver with Master: c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:28.204903 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:54093
I20260929 03:27:28.206573 13699 heartbeater.cc:347] Connected to a master server at 127.6.116.188:41127
I20260929 03:27:28.206619 13699 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.206664 13699 heartbeater.cc:511] Master 127.6.116.188:41127 requested a full tablet report, sending...
I20260929 03:27:28.207010 13520 ts_manager.cc:194] Registered new tserver with Master: c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:28.207726 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:33923
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:46193
--webserver_port=43621
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:28.283329 13707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:28.283522 13707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:28.283558 13707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:28.284749 13707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:28.284807 13707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:28.284850 13707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:28.286276 13707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:33923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=43621
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:28.286499 13707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:28.286711 13707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:28.291282 13712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.291323 13715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.291294 13713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:28.291564 13707 server_base.cc:1034] running on GCE node
I20260929 03:27:28.291812 13707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:28.292030 13707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:28.293174 13707 hybrid_clock.cc:648] HybridClock initialized: now 1790652448293166 us; error 37 us; skew 500 ppm
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:28.295498 13707 init.cc:377] Logged in from keytab as oryx/127.6.116.130@KRBTEST.COM (short username oryx)
I20260929 03:27:28.295921 13707 webserver.cc:460] Webserver started at http://127.6.116.130:43621/ using document root <none> and password file <none>
I20260929 03:27:28.296123 13707 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:28.296195 13707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:28.297370 13707 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:28.298038 13722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.298204 13707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:28.298292 13707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "ebcbc50e1e9444c1ac56025cbb390024"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "9c320a968010dfd17ae10158a11435f9"
server_key_iv: "a2adde79591a2be8700a92c501a58589"
server_key_version: "encryptionkey@0"
I20260929 03:27:28.298569 13707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:28.318404 13707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:28.320205 13707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:28.320351 13707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:28.320541 13707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:28.320839 13707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:28.320883 13707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.320920 13707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:28.320945 13707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.326860 13707 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:33923
I20260929 03:27:28.326903 13835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:33923 every 8 connection(s)
I20260929 03:27:28.327237 13707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
I20260929 03:27:28.338776 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13707
I20260929 03:27:28.338929 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13066
I20260929 03:27:28.341174 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.327826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:50243 (local address 127.6.116.190:46193)
0929 03:27:28.327880 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.327883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.328132 (+ 249us) server_negotiation.cc:407] Connection header received
0929 03:27:28.328686 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.328687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.328699 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.328731 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.329518 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330256 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.331887 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332180 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332223 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.335242 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.335243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.335244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.335252 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.338143 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.340061 (+ 1918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.340062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.340062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.340120 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.340398 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.340399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.340399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.340535 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.340590 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.340964 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.341049 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:27:28.341997 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.327893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:40119 (local address 127.6.116.188:41127)
0929 03:27:28.327939 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.327941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.328137 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:27:28.328689 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.328691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.328706 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.328755 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.329500 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330272 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.331889 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332239 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332307 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.336830 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.336831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.336832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.336839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.338526 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.341088 (+ 2562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.341090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.341090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.341150 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.341352 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.341353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.341353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.341484 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.341550 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.341765 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.341857 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:27:28.342013 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.327813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.6.116.130:40119)
0929 03:27:28.328039 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.328061 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.328568 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.328816 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.328819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.329256 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.329415 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.329419 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330405 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.330406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.331788 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.331791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332362 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.332467 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.333136 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.333141 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.336736 (+ 3595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.338633 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.338634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.338634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.339770 (+ 1136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.341229 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.341230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.341230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.341292 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.341575 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.341575 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.341618 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.341671 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.341794 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260929 03:27:28.342329 13840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.327812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:50243)
0929 03:27:28.327958 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.328058 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.328562 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.328817 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.328821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.329284 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.329416 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.329421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330401 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.330401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.331784 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.331787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332313 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.332465 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.333223 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.333227 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.335123 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.338216 (+ 3093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.338217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.338217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.339982 (+ 1765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.340192 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.340192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.340193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.340312 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.340637 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.340643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.340794 (+ 151us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.341524 (+ 730us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.342240 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":101,"thread_start_us":79,"threads_started":1}
I20260929 03:27:28.342653 13837 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:28.342739 13837 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.342919 13837 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:28.343029 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.328143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:51987 (local address 127.6.116.189:43203)
0929 03:27:28.328193 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.328195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.328328 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:27:28.328685 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.328686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.328699 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.328731 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.329517 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330262 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.331887 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332196 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332232 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.335851 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.335852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.335853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.335860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.337651 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.342207 (+ 4556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.342209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.342209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.342269 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.342461 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.342461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.342462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.342606 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.342656 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.342865 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.342951 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:27:28.343097 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.328092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.6.116.130:51987)
0929 03:27:28.328262 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.328270 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.328572 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.328817 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.328821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.329296 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.329416 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.329421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.330401 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.330401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.331784 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.331787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.332312 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.332313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.332465 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.333084 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.333091 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.335616 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.337769 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.337776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.337802 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.342120 (+ 4318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.342338 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.342338 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.342339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.342399 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.342689 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.342690 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.342730 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.342810 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.342918 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":89,"thread_start_us":58,"threads_started":1}
I20260929 03:27:28.343321 12736 ts_manager.cc:194] Re-registered known tserver with Master: ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923)
I20260929 03:27:28.343485 13838 heartbeater.cc:347] Connected to a master server at 127.6.116.189:43203
I20260929 03:27:28.343547 13838 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.343621 13838 heartbeater.cc:511] Master 127.6.116.189:43203 requested a full tablet report, sending...
I20260929 03:27:28.343768 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:50243
I20260929 03:27:28.343931 13333 ts_manager.cc:194] Registered new tserver with Master: ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923)
I20260929 03:27:28.344128 13836 heartbeater.cc:347] Connected to a master server at 127.6.116.188:41127
I20260929 03:27:28.344179 13836 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.344369 13836 heartbeater.cc:511] Master 127.6.116.188:41127 requested a full tablet report, sending...
I20260929 03:27:28.344636 13520 ts_manager.cc:194] Registered new tserver with Master: ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923)
I20260929 03:27:28.344692 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:46123
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:46193
--webserver_port=35255
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:28.423276 13843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:28.423477 13843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:28.423501 13843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:28.424643 13843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:28.424682 13843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:28.424707 13843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:28.426122 13843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:46123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=35255
--tserver_master_addrs=127.6.116.190:46193,127.6.116.189:43203,127.6.116.188:41127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:28.426363 13843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:28.426563 13843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:28.431010 13849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.431083 13851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:28.431084 13848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:28.431218 13843 server_base.cc:1034] running on GCE node
I20260929 03:27:28.431468 13843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:28.431715 13843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:28.432857 13843 hybrid_clock.cc:648] HybridClock initialized: now 1790652448432846 us; error 24 us; skew 500 ppm
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:28.435141 13843 init.cc:377] Logged in from keytab as oryx/127.6.116.131@KRBTEST.COM (short username oryx)
I20260929 03:27:28.435563 13843 webserver.cc:460] Webserver started at http://127.6.116.131:35255/ using document root <none> and password file <none>
I20260929 03:27:28.435792 13843 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:28.435864 13843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:28.437074 13843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:28.437618 13858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.437915 13843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:28.438002 13843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "3d69323866924d329b10d2a58cc03d8e"
format_stamp: "Formatted at 2026-09-29 03:27:27 on dist-test-slave-z2zg"
server_key: "952c6925c3a1066c70c139737af6d924"
server_key_iv: "f3322646647f88eb338d2aea76cbfc68"
server_key_version: "encryptionkey@0"
I20260929 03:27:28.438319 13843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:28.453183 13843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:28.454720 13843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:28.454859 13843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:28.455104 13843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:28.455389 13843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:28.455438 13843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.455475 13843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:28.455503 13843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:28.461645 13843 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:46123
I20260929 03:27:28.461678 13971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:46123 every 8 connection(s)
I20260929 03:27:28.462036 13843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.189@KRBTEST.COM
I20260929 03:27:28.468959 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 13843
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.188@KRBTEST.COM
Sep 29 03:27:28 dist-test-slave-z2zg krb5kdc[12691](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652448, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:28.473233 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:57755 (local address 127.6.116.189:43203)
0929 03:27:28.462750 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.462752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.462930 (+ 178us) server_negotiation.cc:407] Connection header received
0929 03:27:28.463975 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.463976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.463988 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.464057 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.464932 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465704 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.466737 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.467029 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.467113 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.469463 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.469464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.469464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.469470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.471629 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472114 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.472115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.472116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.472176 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472435 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.472436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.472436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.472572 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.472630 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.473073 (+ 443us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.473149 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:28.473318 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.6.116.131:57755)
0929 03:27:28.462813 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.462849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.463878 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.464119 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.464123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.464495 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.464764 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.464766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465795 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.465795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.466663 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.466664 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.467111 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.467112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.467181 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.467901 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.467906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.469354 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.471707 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.471713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.471715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.472025 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.472248 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.472249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.472366 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.472655 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.472660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.472792 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.472879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.473143 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260929 03:27:28.473856 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:52783 (local address 127.6.116.188:41127)
0929 03:27:28.463004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.463006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.463294 (+ 288us) server_negotiation.cc:407] Connection header received
0929 03:27:28.463992 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.463994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.464006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.464038 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.464853 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465397 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.466734 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.466965 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.467007 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.469988 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.469989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.469990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.469997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.472451 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472837 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.472838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.472838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.472888 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.473104 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.473106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.473106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.473235 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.473304 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.473679 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.473760 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:28.473853 13974 heartbeater.cc:347] Connected to a master server at 127.6.116.189:43203
I20260929 03:27:28.473929 13974 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.474131 13974 heartbeater.cc:511] Master 127.6.116.189:43203 requested a full tablet report, sending...
I20260929 03:27:28.474315 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.6.116.131:52783)
0929 03:27:28.463225 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.463233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.463882 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.464121 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.464123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.464503 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.464744 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.464749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465513 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.465513 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.466625 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.466627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.467124 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.467125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.467182 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.467846 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.467850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.469906 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.472541 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.472543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.472756 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.472958 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.472959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.473010 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.473496 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.473497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.473542 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.474073 (+ 531us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.474216 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260929 03:27:28.474395 12776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:35099 (local address 127.6.116.190:46193)
0929 03:27:28.462727 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.462729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.462929 (+ 200us) server_negotiation.cc:407] Connection header received
0929 03:27:28.463993 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.463994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.464006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.464037 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.464850 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465701 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.466602 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.466884 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.466947 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.470756 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.470758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.470758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.470765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.472751 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.473536 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.473537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.473538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.473614 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.473807 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.473808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.473809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.473927 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.474004 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.474213 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.474286 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:27:28.474534 13333 ts_manager.cc:194] Registered new tserver with Master: 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:28.474773 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.462640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:35099)
0929 03:27:28.462841 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.462850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.463877 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.464121 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.464123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.464506 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:28.464744 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.464749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.465809 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.465810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.466505 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.466507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.466968 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.466969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.467157 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.467833 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:28.467842 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:28.470661 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:28.472829 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.472830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.472830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.473066 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.473685 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:28.473686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:28.473686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:28.473743 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:28.474031 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:28.474031 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:28.474085 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.474143 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.474524 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":320,"spinlock_wait_cycles":6016,"thread_start_us":88,"threads_started":1}
I20260929 03:27:28.475106 13972 heartbeater.cc:347] Connected to a master server at 127.6.116.188:41127
I20260929 03:27:28.475106 13973 heartbeater.cc:347] Connected to a master server at 127.6.116.190:46193
I20260929 03:27:28.475175 13973 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.475247 13973 heartbeater.cc:511] Master 127.6.116.190:46193 requested a full tablet report, sending...
I20260929 03:27:28.475493 12736 ts_manager.cc:194] Re-registered known tserver with Master: 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:28.475971 12736 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:35099
I20260929 03:27:28.475175 13972 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:28.476681 13972 heartbeater.cc:511] Master 127.6.116.188:41127 requested a full tablet report, sending...
I20260929 03:27:28.476922 13520 ts_manager.cc:194] Registered new tserver with Master: 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:28.481407 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.471860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.6.116.189:43203)
0929 03:27:28.473313 (+ 1453us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.473314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.473321 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:28.473338 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.473338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.473346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.473392 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.475671 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.476341 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.477225 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.477429 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.477492 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.478790 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.478791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.478791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.478798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.480344 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.480692 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.480692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.480693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.480743 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.480918 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.480919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.480919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.481040 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.481118 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.481224 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.481309 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1439,"thread_start_us":55,"threads_started":1}
I20260929 03:27:28.481570 13984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.470284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.6.116.190:46193)
0929 03:27:28.470406 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.470408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.471245 (+ 837us) server_negotiation.cc:407] Connection header received
0929 03:27:28.471362 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.471363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.471375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.471441 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.472262 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.472964 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.475873 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.476163 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.476233 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.477573 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.477574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.477574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.477580 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.479264 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.479680 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.479681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.479682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.480448 (+ 766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.480828 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.480829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.480829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.480950 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.481037 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.481365 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.481444 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260929 03:27:28.482005 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.471172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.6.116.188:41127)
0929 03:27:28.471391 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.471394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.471542 (+ 148us) server_negotiation.cc:407] Connection header received
0929 03:27:28.471563 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.471563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.471576 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.471628 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.474765 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.475851 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.477012 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.477348 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.477414 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.479037 (+ 1623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.479056 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.479056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.479062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.480480 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.480852 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.480853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.480853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.480905 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.481084 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.481085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.481085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.481190 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.481302 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.481832 (+ 530us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.481907 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":52,"threads_started":1}
I20260929 03:27:28.483033 12736 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:28.484746 12736 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:28.491389 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.488023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:46123 (local address 127.0.0.1:35062)
0929 03:27:28.488253 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.488262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.488297 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.488682 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.488683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.488685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.488718 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.488719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.489476 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.489477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.490252 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.490253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491164 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.491219 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.491223 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.491285 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.491321 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260929 03:27:28.491391 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.488360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.6.116.131:46123)
0929 03:27:28.488488 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.488492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.488501 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:28.488535 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.488537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.488549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.488629 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.488788 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.489395 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.490367 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491082 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.491225 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.491297 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.491329 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260929 03:27:28.491894 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.487945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33923 (local address 127.0.0.1:58662)
0929 03:27:28.487984 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.487993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.488028 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.488578 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.488578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.488580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.488619 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.488621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.489814 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.489814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.490739 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.490742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491673 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.491733 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.491737 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.491796 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.491833 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:27:28.491911 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.488201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.6.116.130:33923)
0929 03:27:28.488431 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.488436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.488446 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:28.488484 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.488486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.488502 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.488571 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.488696 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.489721 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.490854 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491580 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491670 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.491744 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.491809 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.491841 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260929 03:27:28.492179 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.488006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41905 (local address 127.0.0.1:47328)
0929 03:27:28.488129 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.488138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.488175 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.488883 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.488884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.488886 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.488919 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.488920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.490037 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.490038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.490920 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.490922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491955 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.492017 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.492021 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.492085 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.492118 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260929 03:27:28.492237 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.488209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.6.116.129:41905)
0929 03:27:28.488689 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.488693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.488702 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:28.488738 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.488740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.488753 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.488820 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.488987 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.489962 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491001 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.491789 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.491851 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.491922 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.492119 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.492166 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":85,"threads_started":1}
I20260929 03:27:28.492360 13906 tablet_service.cc:1467] Processing CreateTablet for tablet afa72355a1874b8c976f8fbd7515b75b (DEFAULT_TABLE table=test-table [id=6a5993223a614f96b6d30086f72701ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:28.492512 13770 tablet_service.cc:1467] Processing CreateTablet for tablet afa72355a1874b8c976f8fbd7515b75b (DEFAULT_TABLE table=test-table [id=6a5993223a614f96b6d30086f72701ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:28.492669 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa72355a1874b8c976f8fbd7515b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:28.492832 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa72355a1874b8c976f8fbd7515b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:28.495128 13998 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Bootstrap starting.
I20260929 03:27:28.495609 13999 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Bootstrap starting.
I20260929 03:27:28.495724 13633 tablet_service.cc:1467] Processing CreateTablet for tablet afa72355a1874b8c976f8fbd7515b75b (DEFAULT_TABLE table=test-table [id=6a5993223a614f96b6d30086f72701ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:28.496047 13999 tablet_bootstrap.cc:654] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:28.496040 13633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa72355a1874b8c976f8fbd7515b75b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:28.496280 13999 log.cc:824] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:28.496868 13999 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: No bootstrap required, opened a new log
I20260929 03:27:28.496915 13999 ts_tablet_manager.cc:1397] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:28.498512 14001 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Bootstrap starting.
I20260929 03:27:28.498461 13999 raft_consensus.cc:348] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.498636 13999 raft_consensus.cc:374] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:28.498683 13999 raft_consensus.cc:729] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcbc50e1e9444c1ac56025cbb390024, State: Initialized, Role: FOLLOWER
I20260929 03:27:28.498836 13999 consensus_queue.cc:260] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.499109 13999 ts_tablet_manager.cc:1428] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:28.499185 13837 heartbeater.cc:503] Master 127.6.116.190:46193 was elected leader, sending a full tablet report...
I20260929 03:27:28.499408 14001 tablet_bootstrap.cc:654] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:28.499747 14001 log.cc:824] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:28.499418 13998 tablet_bootstrap.cc:654] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:28.500065 13998 log.cc:824] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:28.500540 14001 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: No bootstrap required, opened a new log
I20260929 03:27:28.500602 14001 ts_tablet_manager.cc:1397] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:28.502522 14001 raft_consensus.cc:348] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.502632 14001 raft_consensus.cc:374] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:28.502653 14001 raft_consensus.cc:729] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c203b981a1e44403820f11e26e54ca4f, State: Initialized, Role: FOLLOWER
I20260929 03:27:28.502772 14001 consensus_queue.cc:260] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.502990 14001 ts_tablet_manager.cc:1428] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:27:28.503072 13700 heartbeater.cc:503] Master 127.6.116.190:46193 was elected leader, sending a full tablet report...
I20260929 03:27:28.503711 13998 tablet_bootstrap.cc:492] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: No bootstrap required, opened a new log
I20260929 03:27:28.503777 13998 ts_tablet_manager.cc:1397] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260929 03:27:28.505180 13998 raft_consensus.cc:348] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.505308 13998 raft_consensus.cc:374] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:28.505352 13998 raft_consensus.cc:729] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d69323866924d329b10d2a58cc03d8e, State: Initialized, Role: FOLLOWER
I20260929 03:27:28.505453 13998 consensus_queue.cc:260] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.505684 13998 ts_tablet_manager.cc:1428] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:28.505744 13973 heartbeater.cc:503] Master 127.6.116.190:46193 was elected leader, sending a full tablet report...
I20260929 03:27:28.575701 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.573405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.6.116.188:41127)
0929 03:27:28.573463 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.573465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.573472 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:28.573495 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.573496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.573513 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.573585 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.573705 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.574253 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.575091 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.575465 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.575526 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.575586 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.575621 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.575647 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:27:28.575706 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.573307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50296)
0929 03:27:28.573376 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.573388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.573435 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.573593 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.573594 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.573595 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.573640 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.573641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.574304 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.574305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.574988 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.574990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.575535 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.575535 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.575577 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.575580 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.575616 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.575652 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 03:27:28.576058 13530 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 03:27:28.576924 13559 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9193e24069244623b206eb66bd093c34. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.577025 13559 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: LEARNER
W20260929 03:27:28.577850 13839 tablet.cc:2369] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:28.590226 14005 raft_consensus.cc:482] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:28.590317 14005 raft_consensus.cc:504] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.590677 14005 leader_election.cc:288] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923), 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:28.593302 13704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.590783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33923 (local address 127.6.116.129:55731)
0929 03:27:28.590826 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.590835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.590870 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.591035 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.591036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.591038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.591132 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.591134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.591753 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.591754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.592475 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.592477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.593033 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.593034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.593108 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.593111 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.593178 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.593215 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:27:28.593318 13995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.590844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:55731 (local address 127.6.116.130:33923)
0929 03:27:28.590911 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.590913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.590922 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:28.590948 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.590948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.590958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.591024 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.591199 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.591678 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.592557 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.592939 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.593034 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.593042 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.593199 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.593237 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:27:28.593392 13997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.590923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:39983 (local address 127.6.116.131:46123)
0929 03:27:28.590964 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.590966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.590977 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:27:28.591024 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.591025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.591034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.591112 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:28.591246 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.591690 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.592591 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.593080 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.593153 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.593161 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.593295 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.593329 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:28.593420 13703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.590858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:46123 (local address 127.6.116.129:39983)
0929 03:27:28.590905 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:28.590915 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:28.590950 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:28.591104 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:28.591105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:28.591106 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:28.591176 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.591178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.591793 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.591794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.592489 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:28.592491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.593155 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.593156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:28.593230 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.593232 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:28.593281 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:27:28.593312 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":2}
I20260929 03:27:28.593752 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa72355a1874b8c976f8fbd7515b75b" candidate_uuid: "c203b981a1e44403820f11e26e54ca4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebcbc50e1e9444c1ac56025cbb390024" is_pre_election: true
I20260929 03:27:28.593902 13790 raft_consensus.cc:2454] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c203b981a1e44403820f11e26e54ca4f in term 0.
I20260929 03:27:28.593946 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa72355a1874b8c976f8fbd7515b75b" candidate_uuid: "c203b981a1e44403820f11e26e54ca4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d69323866924d329b10d2a58cc03d8e" is_pre_election: true
I20260929 03:27:28.594102 13926 raft_consensus.cc:2454] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c203b981a1e44403820f11e26e54ca4f in term 0.
I20260929 03:27:28.594111 13586 leader_election.cc:302] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c203b981a1e44403820f11e26e54ca4f, ebcbc50e1e9444c1ac56025cbb390024; no voters:
I20260929 03:27:28.594270 14005 raft_consensus.cc:2781] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:28.594334 14005 raft_consensus.cc:482] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:28.594352 14005 raft_consensus.cc:3037] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:28.594939 14005 raft_consensus.cc:504] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.595124 14005 leader_election.cc:288] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [CANDIDATE]: Term 1 election: Requested vote from peers ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923), 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123)
I20260929 03:27:28.595233 13790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa72355a1874b8c976f8fbd7515b75b" candidate_uuid: "c203b981a1e44403820f11e26e54ca4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebcbc50e1e9444c1ac56025cbb390024"
I20260929 03:27:28.595309 13790 raft_consensus.cc:3037] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:28.595305 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa72355a1874b8c976f8fbd7515b75b" candidate_uuid: "c203b981a1e44403820f11e26e54ca4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d69323866924d329b10d2a58cc03d8e"
I20260929 03:27:28.595383 13926 raft_consensus.cc:3037] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:28.596007 13790 raft_consensus.cc:2454] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c203b981a1e44403820f11e26e54ca4f in term 1.
I20260929 03:27:28.596122 13926 raft_consensus.cc:2454] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c203b981a1e44403820f11e26e54ca4f in term 1.
I20260929 03:27:28.596159 13586 leader_election.cc:302] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c203b981a1e44403820f11e26e54ca4f, ebcbc50e1e9444c1ac56025cbb390024; no voters:
I20260929 03:27:28.596302 14005 raft_consensus.cc:2781] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:28.596442 14005 raft_consensus.cc:686] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 1 LEADER]: Becoming Leader. State: Replica: c203b981a1e44403820f11e26e54ca4f, State: Running, Role: LEADER
I20260929 03:27:28.596541 14005 consensus_queue.cc:237] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } }
I20260929 03:27:28.597229 12736 catalog_manager.cc:5495] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f reported cstate change: term changed from 0 to 1, leader changed from <none> to c203b981a1e44403820f11e26e54ca4f (127.6.116.129). New cstate: current_term: 1 leader_uuid: "c203b981a1e44403820f11e26e54ca4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:28.597784 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: NON_VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: true } } }
I20260929 03:27:28.597867 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.598506 14012 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: attempting to promote NON_VOTER bee18762a91345ba8363fbd9ab49a782 to VOTER
I20260929 03:27:28.598639 14012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } }
I20260929 03:27:28.599066 13529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 LEARNER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 03:27:28.599124 13342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Refusing update from remote peer 9193e24069244623b206eb66bd093c34: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 03:27:28.599345 13295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260929 03:27:28.599442 13484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 03:27:28.599997 14012 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bee18762a91345ba8363fbd9ab49a782 (127.6.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.600142 13342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bee18762a91345ba8363fbd9ab49a782 (127.6.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.600796 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.600853 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.600921 14012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.600979 14012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.600947 13484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.601033 13484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:28.601150 13530 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bee18762a91345ba8363fbd9ab49a782 (127.6.116.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.602057 14014 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.602056 13559 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9193e24069244623b206eb66bd093c34" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:28.602137 14014 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:27:28.602144 13559 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:27:28.608340 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.606044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.6.116.129:41905)
0929 03:27:28.606086 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.606088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.606113 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:27:28.606194 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.606195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.606219 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.606307 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:28.606426 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.606855 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.607638 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.607828 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.607897 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.608063 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:28.608152 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:28.608238 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.608256 (+ 18us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.608273 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:27:28.609715 13790 raft_consensus.cc:1261] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Refusing update from remote peer c203b981a1e44403820f11e26e54ca4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:28.609736 13926 raft_consensus.cc:1261] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Refusing update from remote peer c203b981a1e44403820f11e26e54ca4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:28.610069 14010 consensus_queue.cc:1035] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:28.610188 14005 consensus_queue.cc:1035] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:28.612679 14017 mvcc.cc:205] Tried to move back new op lower bound from 7334512429503356928 to 7334512429451948032. Current Snapshot: MvccSnapshot[applied={T|T < 7334512429503356928}]
I20260929 03:27:28.613229 14018 mvcc.cc:205] Tried to move back new op lower bound from 7334512429503356928 to 7334512429451948032. Current Snapshot: MvccSnapshot[applied={T|T < 7334512429503356928}]
I20260929 03:27:28.614897 12735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:28.614979 12735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:28.616245 12735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9193e24069244623b206eb66bd093c34: Sending DeleteTablet for 3 replicas of tablet afa72355a1874b8c976f8fbd7515b75b
I20260929 03:27:28.616542 13633 tablet_service.cc:1514] Processing DeleteTablet for tablet afa72355a1874b8c976f8fbd7515b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:28 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:47328
I20260929 03:27:28.616570 13906 tablet_service.cc:1514] Processing DeleteTablet for tablet afa72355a1874b8c976f8fbd7515b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:28 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:35062
I20260929 03:27:28.616551 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet afa72355a1874b8c976f8fbd7515b75b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:28 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:58662
I20260929 03:27:28.616789 14027 tablet_replica.cc:330] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: stopping tablet replica
I20260929 03:27:28.616925 14027 raft_consensus.cc:2229] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:28.616945 14028 tablet_replica.cc:330] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: stopping tablet replica
I20260929 03:27:28.617015 14028 raft_consensus.cc:2229] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:28.617115 14028 raft_consensus.cc:2258] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:28.617115 14027 raft_consensus.cc:2258] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:28.617544 14028 ts_tablet_manager.cc:1905] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:28.617697 14026 tablet_replica.cc:330] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: stopping tablet replica
I20260929 03:27:28.617785 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 12704
I20260929 03:27:28.617797 14026 raft_consensus.cc:2229] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:28.617900 14026 raft_consensus.cc:2258] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:28.618688 14027 ts_tablet_manager.cc:1905] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:28.618753 14026 ts_tablet_manager.cc:1905] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:28.621734 14027 ts_tablet_manager.cc:1918] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:28.621883 14028 ts_tablet_manager.cc:1918] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:28.621984 14028 log.cc:1196] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/afa72355a1874b8c976f8fbd7515b75b
I20260929 03:27:28.622296 14028 ts_tablet_manager.cc:1939] T afa72355a1874b8c976f8fbd7515b75b P 3d69323866924d329b10d2a58cc03d8e: Deleting consensus metadata
I20260929 03:27:28.622429 14027 log.cc:1196] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/afa72355a1874b8c976f8fbd7515b75b
I20260929 03:27:28.622802 14026 ts_tablet_manager.cc:1918] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:28.622946 14026 log.cc:1196] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/afa72355a1874b8c976f8fbd7515b75b
I20260929 03:27:28.622892 14027 ts_tablet_manager.cc:1939] T afa72355a1874b8c976f8fbd7515b75b P ebcbc50e1e9444c1ac56025cbb390024: Deleting consensus metadata
I20260929 03:27:28.623823 14026 ts_tablet_manager.cc:1939] T afa72355a1874b8c976f8fbd7515b75b P c203b981a1e44403820f11e26e54ca4f: Deleting consensus metadata
W20260929 03:27:28.624848 13862 connection.cc:535] server connection from 127.0.0.1:35062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:27:28.624852 13588 connection.cc:535] server connection from 127.0.0.1:47328 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:27:28.625041 13725 connection.cc:535] server connection from 127.0.0.1:58662 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:27:28.628952 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.617387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50304 (local address 127.6.116.188:41127)
0929 03:27:28.617420 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.617422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.618584 (+ 1162us) server_negotiation.cc:407] Connection header received
0929 03:27:28.620286 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.620287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.620297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.620325 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.621016 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.621693 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.623249 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.623564 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.623603 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.625734 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.625735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.625736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.625743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.627549 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.627934 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.627935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.627935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.628000 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.628182 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.628182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.628183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.628294 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.628381 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.628797 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.628870 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:27:28.633494 13384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.626456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.6.116.189:43203)
0929 03:27:28.626497 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.626499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.626769 (+ 270us) server_negotiation.cc:407] Connection header received
0929 03:27:28.626863 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.626863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.626873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.626919 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.627504 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.628183 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.629404 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.629690 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.629760 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.630858 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.630860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.630860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.630866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.632364 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.632737 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.632738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.632738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.632798 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.633004 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.633005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.633005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.633127 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.633215 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.633337 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.633406 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:27:28.634137 13705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:28.627165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.6.116.188:41127)
0929 03:27:28.627202 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:28.627204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:28.627212 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:28.627231 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:28.627232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:28.627241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:28.627294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:28.627966 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.628600 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.629590 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:28.629826 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:28.629894 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:28.631470 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:28.631471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:28.631471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:28.631478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:28.633014 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.633352 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.633353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.633353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.633404 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:28.633628 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:28.633629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:28.633629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:28.633737 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:28.633802 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:28.633975 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:27:28.634043 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:29.613744 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:29.613492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:42019)
0929 03:27:29.613626 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:29.613699 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260929 03:27:29.614061 13973 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:46193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:29.614271 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:29.613927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:48959)
0929 03:27:29.614065 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:29.614226 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260929 03:27:29.614557 13837 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:46193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:29.614727 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:29.614381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:35531)
0929 03:27:29.614574 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:29.614686 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
W20260929 03:27:29.615272 13700 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:46193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:30.116640 14039 raft_consensus.cc:482] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9193e24069244623b206eb66bd093c34)
I20260929 03:27:30.116647 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9193e24069244623b206eb66bd093c34)
I20260929 03:27:30.116746 14039 raft_consensus.cc:504] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } }
I20260929 03:27:30.116784 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } }
I20260929 03:27:30.117012 14039 leader_election.cc:288] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193), b4e8c5b4b0a542a19fdab6f324188b74 (127.6.116.189:43203)
I20260929 03:27:30.117249 14040 leader_election.cc:288] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193), bee18762a91345ba8363fbd9ab49a782 (127.6.116.188:41127)
I20260929 03:27:30.117312 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42934)
0929 03:27:30.117213 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.117262 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":44,"threads_started":1}
I20260929 03:27:30.117517 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42932)
0929 03:27:30.117434 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.117479 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":47,"threads_started":1}
W20260929 03:27:30.117584 13320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
W20260929 03:27:30.117645 13507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:30.118012 14042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42944)
0929 03:27:30.117969 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.117989 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 03:27:30.118083 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42954)
0929 03:27:30.118017 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.118049 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:27:30.118130 13507 leader_election.cc:334] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193): Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
W20260929 03:27:30.118161 13320 leader_election.cc:334] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193): Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:30.120234 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.6.116.188:41127)
0929 03:27:30.117547 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.117549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.117555 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:30.117577 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.117578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.117588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.117822 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.117868 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.118491 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.119532 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.119977 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.120036 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.120047 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.120164 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.120195 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":46,"threads_started":1}
I20260929 03:27:30.120250 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:41127 (local address 127.0.0.1:50320)
0929 03:27:30.117375 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.117384 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.117434 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.117713 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.117714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.117717 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.117777 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.117778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.118561 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.118561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.119466 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.119468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.120042 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.120043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.120101 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.120105 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.120155 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.120191 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260929 03:27:30.120440 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bee18762a91345ba8363fbd9ab49a782" is_pre_election: true
I20260929 03:27:30.120534 13530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4e8c5b4b0a542a19fdab6f324188b74 in term 1.
I20260929 03:27:30.120690 13317 leader_election.cc:302] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4e8c5b4b0a542a19fdab6f324188b74, bee18762a91345ba8363fbd9ab49a782; no voters: 9193e24069244623b206eb66bd093c34
I20260929 03:27:30.120803 14040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 03:27:30.120862 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9193e24069244623b206eb66bd093c34)
I20260929 03:27:30.120879 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43203 (local address 127.0.0.1:54578)
0929 03:27:30.117467 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.117478 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.117869 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.118147 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.118148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.118150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.118262 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.118265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.119088 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.119089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.119960 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.119962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.120634 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.120635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.120696 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.120699 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.120763 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.120800 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":44,"threads_started":1}
I20260929 03:27:30.120888 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:27:30.120939 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.117169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.6.116.189:43203)
0929 03:27:30.117274 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.117276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.117531 (+ 255us) server_negotiation.cc:407] Connection header received
0929 03:27:30.117966 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.117967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.117977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.118037 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.118344 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.118982 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.120040 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.120570 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.120633 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.120642 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.120767 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.120801 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":84,"thread_start_us":32,"threads_started":1}
I20260929 03:27:30.121088 13342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bee18762a91345ba8363fbd9ab49a782" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" is_pre_election: true
I20260929 03:27:30.121670 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } }
I20260929 03:27:30.121781 13342 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bee18762a91345ba8363fbd9ab49a782 in current term 2: Already voted for candidate b4e8c5b4b0a542a19fdab6f324188b74 in this term.
I20260929 03:27:30.121811 14040 leader_election.cc:288] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 election: Requested vote from peers 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193), bee18762a91345ba8363fbd9ab49a782 (127.6.116.188:41127)
I20260929 03:27:30.121981 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.121904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42966)
0929 03:27:30.121935 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.121963 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 03:27:30.121963 13530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bee18762a91345ba8363fbd9ab49a782"
I20260929 03:27:30.122015 13504 leader_election.cc:302] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bee18762a91345ba8363fbd9ab49a782; no voters: 9193e24069244623b206eb66bd093c34, b4e8c5b4b0a542a19fdab6f324188b74
I20260929 03:27:30.122022 13530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:27:30.122332 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.122241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42968)
0929 03:27:30.122289 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.122317 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:27:30.122400 13320 leader_election.cc:334] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193): Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
I20260929 03:27:30.122861 13530 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4e8c5b4b0a542a19fdab6f324188b74 in term 2.
I20260929 03:27:30.122992 14039 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:27:30.123144 13317 leader_election.cc:302] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4e8c5b4b0a542a19fdab6f324188b74, bee18762a91345ba8363fbd9ab49a782; no voters: 9193e24069244623b206eb66bd093c34
I20260929 03:27:30.123252 14040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:27:30.123377 14040 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [term 2 LEADER]: Becoming Leader. State: Replica: b4e8c5b4b0a542a19fdab6f324188b74, State: Running, Role: LEADER
I20260929 03:27:30.123456 14040 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } }
I20260929 03:27:30.123772 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e8c5b4b0a542a19fdab6f324188b74. Latest consensus state: current_term: 2 leader_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:30.123876 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:30.124035 14051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:30.124401 14051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:30.124454 14051 catalog_manager.cc:1259] Loaded cluster ID: d44b12c1a1c34da7b48b6b58c84eda7b
I20260929 03:27:30.124475 14051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:30.124583 14051 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:30.124631 14051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Loaded TSK: 0
I20260929 03:27:30.124715 14051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:30.167233 13333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:30.168782 13333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:30.169631 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.169556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42992)
0929 03:27:30.169584 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.169610 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260929 03:27:30.169649 13530 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [term 2 FOLLOWER]: Refusing update from remote peer b4e8c5b4b0a542a19fdab6f324188b74: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 03:27:30.169871 14049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 03:27:30.170095 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.169895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.0.0.1:42994)
0929 03:27:30.170008 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.170067 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260929 03:27:30.170183 13320 consensus_peers.cc:487] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 -> Peer 9193e24069244623b206eb66bd093c34 (127.6.116.190:46193): Couldn't send request to peer 9193e24069244623b206eb66bd093c34. Status: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 03:27:30.170497 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e8c5b4b0a542a19fdab6f324188b74. Latest consensus state: current_term: 2 leader_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:30.170588 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:27:30.171007 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:30.171126 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:30.171152 14039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "9193e24069244623b206eb66bd093c34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 46193 } } peers { permanent_uuid: "b4e8c5b4b0a542a19fdab6f324188b74" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43203 } attrs { promote: false } } peers { permanent_uuid: "bee18762a91345ba8363fbd9ab49a782" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 41127 } attrs { promote: false } } }
I20260929 03:27:30.171224 14039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bee18762a91345ba8363fbd9ab49a782 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:27:30.175662 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33923 (local address 127.0.0.1:58676)
0929 03:27:30.172768 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.172790 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.172839 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.173023 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.173024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.173027 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.173073 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.173074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.173716 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.173718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.174611 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.174613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.175431 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.175432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.175492 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.175497 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.175545 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.175585 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260929 03:27:30.175729 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.6.116.131:46123)
0929 03:27:30.172867 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.172869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.172878 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:30.172903 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.172904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.172915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.173123 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.173161 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.173881 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.174847 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.175449 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.175516 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.175525 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.175630 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.175661 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":53,"threads_started":1}
I20260929 03:27:30.175742 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:46123 (local address 127.0.0.1:35072)
0929 03:27:30.172610 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.172620 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.172654 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.173005 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.173006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.173007 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.173058 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.173060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.173962 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.173963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.174752 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.174754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.175512 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.175513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.175566 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.175569 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.175621 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.175649 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":49}
I20260929 03:27:30.175933 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.6.116.130:33923)
0929 03:27:30.172864 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.172866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.172873 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:30.172907 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.172908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.172921 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.173136 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.173174 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.173641 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.174699 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.175361 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.175434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.175445 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.175820 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.175867 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260929 03:27:30.176178 13770 tablet_service.cc:1467] Processing CreateTablet for tablet f01ee3bc3666426a989a4266377e19bf (DEFAULT_TABLE table=test-table [id=c8cc3a6b6370459aa9a687c4e420019b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:30.176185 13906 tablet_service.cc:1467] Processing CreateTablet for tablet f01ee3bc3666426a989a4266377e19bf (DEFAULT_TABLE table=test-table [id=c8cc3a6b6370459aa9a687c4e420019b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:30.176331 13770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01ee3bc3666426a989a4266377e19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:30.176334 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01ee3bc3666426a989a4266377e19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:30.176367 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41905 (local address 127.0.0.1:47358)
0929 03:27:30.173018 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.173026 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.173063 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.173299 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.173300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.173302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.173335 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.173336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.174139 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.174140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.175199 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.175201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.176121 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.176122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.176177 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.176182 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.176239 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.176279 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":249,"thread_start_us":87,"threads_started":1}
I20260929 03:27:30.176378 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.172769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.6.116.129:41905)
0929 03:27:30.173151 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.173154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.173160 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:30.173182 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.173183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.173195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.173267 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.173401 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.174068 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.175277 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.176038 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.176117 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.176129 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.176251 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.176310 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":61,"threads_started":1}
I20260929 03:27:30.176662 13633 tablet_service.cc:1467] Processing CreateTablet for tablet f01ee3bc3666426a989a4266377e19bf (DEFAULT_TABLE table=test-table [id=c8cc3a6b6370459aa9a687c4e420019b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:30.176755 13633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f01ee3bc3666426a989a4266377e19bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:30.177436 14062 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Bootstrap starting.
I20260929 03:27:30.177731 14062 tablet_bootstrap.cc:654] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:30.178007 14063 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Bootstrap starting.
I20260929 03:27:30.178373 14063 tablet_bootstrap.cc:654] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:30.178654 14065 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Bootstrap starting.
I20260929 03:27:30.178663 14062 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: No bootstrap required, opened a new log
I20260929 03:27:30.178711 14062 ts_tablet_manager.cc:1397] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:30.178834 14062 raft_consensus.cc:348] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.178908 14063 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: No bootstrap required, opened a new log
I20260929 03:27:30.178911 14062 raft_consensus.cc:374] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:30.178933 14062 raft_consensus.cc:729] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebcbc50e1e9444c1ac56025cbb390024, State: Initialized, Role: FOLLOWER
I20260929 03:27:30.178946 14063 ts_tablet_manager.cc:1397] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:30.178987 14062 consensus_queue.cc:260] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.179069 14065 tablet_bootstrap.cc:654] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:30.179077 14063 raft_consensus.cc:348] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.179159 14063 raft_consensus.cc:374] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:30.179152 14062 ts_tablet_manager.cc:1428] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:30.179575 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.179394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:41731)
0929 03:27:30.179503 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.179538 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":69,"threads_started":1}
I20260929 03:27:30.179186 14063 raft_consensus.cc:729] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d69323866924d329b10d2a58cc03d8e, State: Initialized, Role: FOLLOWER
I20260929 03:27:30.179672 14065 tablet_bootstrap.cc:492] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: No bootstrap required, opened a new log
I20260929 03:27:30.179718 14065 ts_tablet_manager.cc:1397] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:30.179698 14063 consensus_queue.cc:260] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.179776 13838 heartbeater.cc:503] Master 127.6.116.189:43203 was elected leader, sending a full tablet report...
I20260929 03:27:30.179818 14065 raft_consensus.cc:348] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.179862 14063 ts_tablet_manager.cc:1428] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:30.179879 14065 raft_consensus.cc:374] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:30.179898 14065 raft_consensus.cc:729] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c203b981a1e44403820f11e26e54ca4f, State: Initialized, Role: FOLLOWER
I20260929 03:27:30.179944 14065 consensus_queue.cc:260] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.180089 14065 ts_tablet_manager.cc:1428] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:30.180469 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.180216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:51615)
0929 03:27:30.180401 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.180434 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":72,"threads_started":1}
I20260929 03:27:30.180763 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.180691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:36461)
0929 03:27:30.180721 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.180743 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 03:27:30.180902 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.180794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:60945)
0929 03:27:30.180842 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.180869 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:27:30.181087 13701 heartbeater.cc:503] Master 127.6.116.189:43203 was elected leader, sending a full tablet report...
I20260929 03:27:30.181344 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.180350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:46595)
0929 03:27:30.181288 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.181317 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":912,"thread_start_us":69,"threads_started":1}
W20260929 03:27:30.193065 13702 tablet.cc:2369] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:30.214102 13975 tablet.cc:2369] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:30.242388 14067 raft_consensus.cc:482] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:30.242440 14067 raft_consensus.cc:504] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.242724 14067 leader_election.cc:288] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123), c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:30.245476 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.242887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55117 (local address 127.6.116.129:41905)
0929 03:27:30.242926 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.242928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.242936 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:30.242984 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.242985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.242994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.243079 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.243227 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.243717 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.244680 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.245156 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.245234 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.245242 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.245389 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.245421 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:30.245471 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.242841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41905 (local address 127.6.116.130:55117)
0929 03:27:30.242879 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.242888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.242935 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.243076 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.243077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.243079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.243145 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.243146 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.243786 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.243787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.244604 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.244619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.245264 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.245265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.245322 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.245325 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.245376 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.245403 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":40}
I20260929 03:27:30.245476 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.242901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:52813 (local address 127.6.116.131:46123)
0929 03:27:30.242946 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.242948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.243132 (+ 184us) server_negotiation.cc:407] Connection header received
0929 03:27:30.243163 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.243164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.243174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.243225 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:30.243347 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.243741 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.244633 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.245106 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.245182 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.245210 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.245382 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.245418 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:30.245491 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.242841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:46123 (local address 127.6.116.130:52813)
0929 03:27:30.243068 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.243077 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.243123 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.243226 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:30.243227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:30.243228 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:30.243284 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.243285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.243844 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.243845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.244571 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:30.244574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.245179 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.245180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:30.245290 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.245294 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:30.245354 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:27:30.245395 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":75,"thread_start_us":87,"threads_started":1}
I20260929 03:27:30.245718 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01ee3bc3666426a989a4266377e19bf" candidate_uuid: "ebcbc50e1e9444c1ac56025cbb390024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d69323866924d329b10d2a58cc03d8e" is_pre_election: true
I20260929 03:27:30.245784 13926 raft_consensus.cc:2454] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcbc50e1e9444c1ac56025cbb390024 in term 0.
I20260929 03:27:30.245877 13653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01ee3bc3666426a989a4266377e19bf" candidate_uuid: "ebcbc50e1e9444c1ac56025cbb390024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c203b981a1e44403820f11e26e54ca4f" is_pre_election: true
I20260929 03:27:30.245932 13725 leader_election.cc:302] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d69323866924d329b10d2a58cc03d8e, ebcbc50e1e9444c1ac56025cbb390024; no voters:
I20260929 03:27:30.245980 13653 raft_consensus.cc:2454] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebcbc50e1e9444c1ac56025cbb390024 in term 0.
I20260929 03:27:30.246048 14067 raft_consensus.cc:2781] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:30.246085 14067 raft_consensus.cc:482] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:30.246106 14067 raft_consensus.cc:3037] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:30.246685 14067 raft_consensus.cc:504] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.246799 14067 leader_election.cc:288] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [CANDIDATE]: Term 1 election: Requested vote from peers 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123), c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905)
I20260929 03:27:30.246914 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01ee3bc3666426a989a4266377e19bf" candidate_uuid: "ebcbc50e1e9444c1ac56025cbb390024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d69323866924d329b10d2a58cc03d8e"
I20260929 03:27:30.246958 13653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f01ee3bc3666426a989a4266377e19bf" candidate_uuid: "ebcbc50e1e9444c1ac56025cbb390024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c203b981a1e44403820f11e26e54ca4f"
I20260929 03:27:30.246997 13926 raft_consensus.cc:3037] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:30.247033 13653 raft_consensus.cc:3037] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:30.247557 13926 raft_consensus.cc:2454] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcbc50e1e9444c1ac56025cbb390024 in term 1.
I20260929 03:27:30.247597 13653 raft_consensus.cc:2454] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebcbc50e1e9444c1ac56025cbb390024 in term 1.
I20260929 03:27:30.247741 13725 leader_election.cc:302] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d69323866924d329b10d2a58cc03d8e, ebcbc50e1e9444c1ac56025cbb390024; no voters:
I20260929 03:27:30.247833 14067 raft_consensus.cc:2781] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:30.247952 14067 raft_consensus.cc:686] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 1 LEADER]: Becoming Leader. State: Replica: ebcbc50e1e9444c1ac56025cbb390024, State: Running, Role: LEADER
I20260929 03:27:30.248016 14067 consensus_queue.cc:237] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } }
I20260929 03:27:30.248306 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.248203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:44837)
0929 03:27:30.248251 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.248283 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:27:30.248646 13332 catalog_manager.cc:5495] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130). New cstate: current_term: 1 leader_uuid: "ebcbc50e1e9444c1ac56025cbb390024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebcbc50e1e9444c1ac56025cbb390024" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:30.267102 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.264918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.6.116.130:33923)
0929 03:27:30.264957 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:30.264959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:30.264967 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:30.264997 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:30.264998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:30.265013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:30.265073 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:30.265172 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.265644 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.266372 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:30.266569 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:30.266652 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:30.266799 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:30.266890 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:30.266962 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:30.267002 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:27:30.267019 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:27:30.267930 13926 raft_consensus.cc:1261] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Refusing update from remote peer ebcbc50e1e9444c1ac56025cbb390024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:30.268055 13653 raft_consensus.cc:1261] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Refusing update from remote peer ebcbc50e1e9444c1ac56025cbb390024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:30.268177 14067 consensus_queue.cc:1035] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d69323866924d329b10d2a58cc03d8e" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 46123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:30.268294 14075 consensus_queue.cc:1035] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c203b981a1e44403820f11e26e54ca4f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:30.268657 13974 heartbeater.cc:503] Master 127.6.116.189:43203 was elected leader, sending a full tablet report...
I20260929 03:27:30.268949 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.268856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:34881)
0929 03:27:30.268886 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.268920 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260929 03:27:30.269816 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.269154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:55429)
0929 03:27:30.269656 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.269791 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485}
I20260929 03:27:30.270262 14074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.269991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.130:46409)
0929 03:27:30.270205 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.270235 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260929 03:27:30.270407 14073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.270332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.131:35877)
0929 03:27:30.270364 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.270385 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:27:30.270525 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:30.270347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:46193 (local address 127.6.116.129:37113)
0929 03:27:30.270480 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.270504 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260929 03:27:30.270826 14080 mvcc.cc:205] Tried to move back new op lower bound from 7334512436295757824 to 7334512436216225792. Current Snapshot: MvccSnapshot[applied={T|T < 7334512436295757824 or (T in {7334512436295757824})}]
I20260929 03:27:30.272562 13332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:30.272641 13332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:30.273707 13332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4e8c5b4b0a542a19fdab6f324188b74: Sending DeleteTablet for 3 replicas of tablet f01ee3bc3666426a989a4266377e19bf
I20260929 03:27:30.273947 13906 tablet_service.cc:1514] Processing DeleteTablet for tablet f01ee3bc3666426a989a4266377e19bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:30 UTC) from {username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:35072
I20260929 03:27:30.273989 13633 tablet_service.cc:1514] Processing DeleteTablet for tablet f01ee3bc3666426a989a4266377e19bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:30 UTC) from {username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:47358
I20260929 03:27:30.274142 14088 tablet_replica.cc:330] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: stopping tablet replica
I20260929 03:27:30.274143 14087 tablet_replica.cc:330] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: stopping tablet replica
I20260929 03:27:30.274181 14088 raft_consensus.cc:2229] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:30.274183 14087 raft_consensus.cc:2229] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:30.274217 14088 raft_consensus.cc:2258] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:30.274217 14087 raft_consensus.cc:2258] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:30.274235 13770 tablet_service.cc:1514] Processing DeleteTablet for tablet f01ee3bc3666426a989a4266377e19bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:30 UTC) from {username='oryx', principal='oryx/127.6.116.189@KRBTEST.COM'} at 127.0.0.1:58676
I20260929 03:27:30.274488 14088 ts_tablet_manager.cc:1905] T f01ee3bc3666426a989a4266377e19bf P c203b981a1e44403820f11e26e54ca4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:30.274613 14087 ts_tablet_manager.cc:1905] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:30.274619 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13481
I20260929 03:27:30.276029 14089 tablet_replica.cc:330] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: stopping tablet replica
I20260929 03:27:30.276096 14089 raft_consensus.cc:2229] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:30.276180 14089 raft_consensus.cc:2258] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:30.279505 14089 ts_tablet_manager.cc:1905] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:30.281333 14089 ts_tablet_manager.cc:1918] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:30.281394 14089 log.cc:1196] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/f01ee3bc3666426a989a4266377e19bf
I20260929 03:27:30.281629 14089 ts_tablet_manager.cc:1939] T f01ee3bc3666426a989a4266377e19bf P ebcbc50e1e9444c1ac56025cbb390024: Deleting consensus metadata
W20260929 03:27:30.281863 14061 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:30.281564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41905 (local address 127.0.0.1:47368)
0929 03:27:30.281595 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:30.281605 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:30.281651 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:30.281821 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:41905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 03:27:30.281916 13317 catalog_manager.cc:4841] TS ebcbc50e1e9444c1ac56025cbb390024 (127.6.116.130:33923): tablet f01ee3bc3666426a989a4266377e19bf (table test-table [id=c8cc3a6b6370459aa9a687c4e420019b]) successfully deleted
W20260929 03:27:30.281976 13320 catalog_manager.cc:4584] TS c203b981a1e44403820f11e26e54ca4f (127.6.116.129:41905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f01ee3bc3666426a989a4266377e19bf: Network error: Client connection negotiation failed: client connection to 127.6.116.129:41905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:27:30.282341 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13707
I20260929 03:27:30.283622 14087 ts_tablet_manager.cc:1918] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:30.283691 14087 log.cc:1196] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/f01ee3bc3666426a989a4266377e19bf
I20260929 03:27:30.283877 14087 ts_tablet_manager.cc:1939] T f01ee3bc3666426a989a4266377e19bf P 3d69323866924d329b10d2a58cc03d8e: Deleting consensus metadata
I20260929 03:27:30.284232 13319 catalog_manager.cc:4841] TS 3d69323866924d329b10d2a58cc03d8e (127.6.116.131:46123): tablet f01ee3bc3666426a989a4266377e19bf (table test-table [id=c8cc3a6b6370459aa9a687c4e420019b]) successfully deleted
I20260929 03:27:30.287722 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13843
I20260929 03:27:30.291887 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13200
I20260929 03:27:30.296607 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 13387
2026-09-29T03:27:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4795 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:30 dist-test-slave-z2zg krb5kdc[14092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:30 dist-test-slave-z2zg krb5kdc[14092](info): set up 2 sockets
Sep 29 03:27:30 dist-test-slave-z2zg krb5kdc[14092](info): commencing operation
krb5kdc: starting...
W20260929 03:27:31.317250 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab.
Entry for principal oryx/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab.
I20260929 03:27:31.424850 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34103
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab
--principal=oryx/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:34103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:31.500833 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:31.501034 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:31.501070 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:31.502146 14105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:31.502204 14105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 03:27:31.502226 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:31.502249 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:31.502274 14105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:31.502295 14105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:31.503783 14105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:34103
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.190.keytab
--principal=oryx/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:31.504037 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:31.504262 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:31.508929 14111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:31.508963 14110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.509042 14105 server_base.cc:1034] running on GCE node
W20260929 03:27:31.508962 14113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.509408 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:31.509634 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:31.510789 14105 hybrid_clock.cc:648] HybridClock initialized: now 1790652451510778 us; error 29 us; skew 500 ppm
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:31.512975 14105 init.cc:377] Logged in from keytab as oryx/127.6.116.190@KRBTEST.COM (short username oryx)
I20260929 03:27:31.513379 14105 webserver.cc:460] Webserver started at http://127.6.116.190:46821/ using document root <none> and password file <none>
I20260929 03:27:31.513572 14105 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:31.513638 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:31.513773 14105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:31.514684 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "6d493630b5504a9a8379931cec67efcc"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "8ccde1fa12d2e2cc681142688f36364f"
server_key_iv: "ebfefebb8f549975adbb80664724d665"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.515007 14105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "6d493630b5504a9a8379931cec67efcc"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "8ccde1fa12d2e2cc681142688f36364f"
server_key_iv: "ebfefebb8f549975adbb80664724d665"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.516481 14105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:31.517206 14120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:31.517385 14105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:31.517494 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "6d493630b5504a9a8379931cec67efcc"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "8ccde1fa12d2e2cc681142688f36364f"
server_key_iv: "ebfefebb8f549975adbb80664724d665"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.517573 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:31.532711 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:31.534227 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:31.534370 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:31.538863 14105 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:34103
I20260929 03:27:31.538913 14182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:34103 every 8 connection(s)
I20260929 03:27:31.539225 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:31.539793 14183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:31.541853 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Bootstrap starting.
I20260929 03:27:31.542398 14183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:31.542709 14183 log.cc:824] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:31.543345 14183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: No bootstrap required, opened a new log
I20260929 03:27:31.545238 14183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } }
I20260929 03:27:31.545380 14183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:31.545423 14183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d493630b5504a9a8379931cec67efcc, State: Initialized, Role: FOLLOWER
I20260929 03:27:31.545574 14183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } }
I20260929 03:27:31.545655 14183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:31.545703 14183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:31.545749 14183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:31.546236 14183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } }
I20260929 03:27:31.546339 14183 leader_election.cc:302] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d493630b5504a9a8379931cec67efcc; no voters:
I20260929 03:27:31.546504 14183 leader_election.cc:288] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:31.546536 14186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:31.546715 14186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [term 1 LEADER]: Becoming Leader. State: Replica: 6d493630b5504a9a8379931cec67efcc, State: Running, Role: LEADER
I20260929 03:27:31.546741 14183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:31.546806 14186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } }
I20260929 03:27:31.547183 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d493630b5504a9a8379931cec67efcc. Latest consensus state: current_term: 1 leader_uuid: "6d493630b5504a9a8379931cec67efcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } } }
I20260929 03:27:31.547297 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:31.547180 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d493630b5504a9a8379931cec67efcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d493630b5504a9a8379931cec67efcc" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34103 } } }
I20260929 03:27:31.547432 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:31.547741 14195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:31.548199 14195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:31.548738 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14105
I20260929 03:27:31.548840 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:31.549026 6610 external_mini_cluster.cc:1428] Setting key a6e7cbd038f8c8e6423b6842a51c1c65
I20260929 03:27:31.549544 14195 catalog_manager.cc:1347] Generated new cluster ID: 057d24b3902a4aaf925e4a0257056edc
I20260929 03:27:31.549602 14195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:31.561795 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.549415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.6.116.190:34103)
0929 03:27:31.551170 (+ 1755us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:31.551179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:31.551210 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:31.551317 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:31.551323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:31.551373 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:31.551514 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:31.552181 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.554201 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.555194 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.555857 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.555962 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.558555 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:31.558558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:31.558559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:31.558569 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:31.560325 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.560720 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.560722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.560723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.560813 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.560998 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.560998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.560998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.561248 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:31.561355 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:31.561474 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 03:27:31.561588 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1711,"thread_start_us":62,"threads_started":1}
I20260929 03:27:31.565161 14195 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:31.565634 14195 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:31.575702 14195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Generated new TSK 0
I20260929 03:27:31.575847 14195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
I20260929 03:27:31.605472 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34103
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:31.708266 14209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:31.708437 14209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:31.708459 14209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:31.709568 14209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:31.709609 14209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:31.709645 14209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:31.709679 14209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:31.711108 14209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:31.711323 14209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:31.711519 14209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:31.716086 14214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.716230 14209 server_base.cc:1034] running on GCE node
W20260929 03:27:31.716086 14215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:31.716143 14217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.716508 14209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:31.716748 14209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:31.717921 14209 hybrid_clock.cc:648] HybridClock initialized: now 1790652451717904 us; error 33 us; skew 500 ppm
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:31.720335 14209 init.cc:377] Logged in from keytab as oryx/127.6.116.129@KRBTEST.COM (short username oryx)
I20260929 03:27:31.720714 14209 webserver.cc:460] Webserver started at http://127.6.116.129:33421/ using document root <none> and password file <none>
I20260929 03:27:31.720888 14209 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:31.720925 14209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:31.721040 14209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:31.721958 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "9bd2b1b740bd99435d647d957131ca69"
server_key_iv: "d6118621763f0d8cbdf8687736149534"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.722255 14209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "9bd2b1b740bd99435d647d957131ca69"
server_key_iv: "d6118621763f0d8cbdf8687736149534"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.723431 14209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:31.724071 14224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:31.724244 14209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:31.724325 14209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "9bd2b1b740bd99435d647d957131ca69"
server_key_iv: "d6118621763f0d8cbdf8687736149534"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.724392 14209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:31.735265 14209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:31.737327 14209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:31.737468 14209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:31.737886 14209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:31.737921 14209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:31.737941 14209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:31.737979 14209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:31.745095 14209 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:41327
I20260929 03:27:31.745393 14339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:41327 every 8 connection(s)
I20260929 03:27:31.745486 14209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:31.747684 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.738312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:42397 (local address 127.6.116.190:34103)
0929 03:27:31.738376 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:31.738378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:31.738593 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:27:31.739291 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:31.739291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:31.739303 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:31.739361 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:31.740243 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.740746 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.742367 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.742568 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.742647 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.744831 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:31.744833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:31.744833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:31.744841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:31.746481 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.746883 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.746883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.746884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.746934 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.747184 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.747185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.747185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.747290 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:31.747337 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:31.747559 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 03:27:31.747626 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:31.747895 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.738291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.129:42397)
0929 03:27:31.738463 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:31.738508 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:31.739152 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:31.739441 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:31.739446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:31.739862 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:31.740150 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.740156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.740813 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.740814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.742266 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.742269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.742670 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.742671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.742788 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.743272 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:31.743287 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:31.744726 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:31.746559 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.746565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.746567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.746821 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.746977 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.746977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.746977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.747119 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.747331 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:31.747336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:31.747445 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 03:27:31.747523 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:31.747660 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260929 03:27:31.748360 14340 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34103
I20260929 03:27:31.748440 14340 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:31.748606 14340 heartbeater.cc:511] Master 127.6.116.190:34103 requested a full tablet report, sending...
I20260929 03:27:31.749059 14137 ts_manager.cc:194] Registered new tserver with Master: e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327)
I20260929 03:27:31.749683 14137 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:42397
I20260929 03:27:31.750059 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14209
I20260929 03:27:31.750185 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:31.750365 6610 external_mini_cluster.cc:1428] Setting key b1f89b9d6a97b369774e57bf5b1be043
I20260929 03:27:31.755517 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.749293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:44133 (local address 127.6.116.190:34103)
0929 03:27:31.749332 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:31.749333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:31.749943 (+ 610us) server_negotiation.cc:407] Connection header received
0929 03:27:31.750007 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:31.750008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:31.750017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:31.750059 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:31.750547 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.751131 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.752110 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.752312 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.752380 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.753196 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:31.753197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:31.753198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:31.753204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:31.754444 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.754767 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.754768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.754768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.754817 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.755002 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.755003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.755003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.755134 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:31.755198 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:31.755360 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:27:31.755433 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:31.755533 14346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.749240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.129:44133)
0929 03:27:31.749893 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:31.749903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:31.749955 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:31.750096 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:31.750097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:31.750352 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:31.750447 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.750449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.751212 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.751213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.752034 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.752036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.752378 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.752379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.752445 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.752843 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:31.752848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:31.753101 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:31.754524 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.754525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.754525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.754688 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.754882 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.754882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.754883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.754938 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.755202 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:31.755203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:31.755243 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:27:31.755337 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 03:27:31.755433 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":57,"threads_started":1}
WARNING: no policy specified for oryx/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
I20260929 03:27:31.773479 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34103
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:31.849696 14349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:31.849907 14349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:31.849946 14349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:31.851017 14349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:31.851104 14349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:31.851128 14349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:31.851146 14349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:31.852483 14349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:31.852649 14349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:31.852794 14349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:31.857199 14355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:31.857261 14354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.857352 14349 server_base.cc:1034] running on GCE node
W20260929 03:27:31.857261 14357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:31.857596 14349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:31.857851 14349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:31.859169 14349 hybrid_clock.cc:648] HybridClock initialized: now 1790652451859147 us; error 25 us; skew 500 ppm
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:31.861460 14349 init.cc:377] Logged in from keytab as oryx/127.6.116.130@KRBTEST.COM (short username oryx)
I20260929 03:27:31.861845 14349 webserver.cc:460] Webserver started at http://127.6.116.130:40789/ using document root <none> and password file <none>
I20260929 03:27:31.862046 14349 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:31.862112 14349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:31.862234 14349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:31.863253 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "52105dbd97c2cd68fd33365d2a586b48"
server_key_iv: "d74586db8451167b8d9280305056a999"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.863606 14349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "52105dbd97c2cd68fd33365d2a586b48"
server_key_iv: "d74586db8451167b8d9280305056a999"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.864820 14349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:31.865477 14364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:31.865648 14349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:31.865737 14349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
format_stamp: "Formatted at 2026-09-29 03:27:31 on dist-test-slave-z2zg"
server_key: "52105dbd97c2cd68fd33365d2a586b48"
server_key_iv: "d74586db8451167b8d9280305056a999"
server_key_version: "encryptionkey@0"
I20260929 03:27:31.865808 14349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:31.894709 14349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:31.896364 14349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:31.896521 14349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:31.896953 14349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:31.897009 14349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:31.897049 14349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:31.897086 14349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:31 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:31.903983 14349 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:37211
I20260929 03:27:31.904111 14479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:37211 every 8 connection(s)
I20260929 03:27:31.904376 14349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:31.906234 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.897413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:49239 (local address 127.6.116.190:34103)
0929 03:27:31.897475 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:31.897478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:31.897677 (+ 199us) server_negotiation.cc:407] Connection header received
0929 03:27:31.898087 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:31.898089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:31.898102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:31.898184 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:31.898929 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.899746 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.900711 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.901035 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.901112 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.903491 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:31.903492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:31.903493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:31.903500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:31.905107 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.905455 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.905455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.905456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.905518 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.905761 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.905762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.905762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.905853 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:31.905901 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:31.906106 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:27:31.906163 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:27:31.906402 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.897385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.130:49239)
0929 03:27:31.897547 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:31.897582 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:31.897999 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:31.898227 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:31.898229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:31.898556 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:31.898829 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.898834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.899865 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.899866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.900645 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.900647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.901137 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.901138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.901252 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.901698 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:31.901712 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:31.903394 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:31.905179 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.905183 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.905184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.905394 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.905573 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.905574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.905575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.905708 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.905904 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:31.905908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:31.906005 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 03:27:31.906081 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:31.906187 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260929 03:27:31.906849 14480 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34103
I20260929 03:27:31.906924 14480 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:31.907121 14480 heartbeater.cc:511] Master 127.6.116.190:34103 requested a full tablet report, sending...
I20260929 03:27:31.907517 14136 ts_manager.cc:194] Registered new tserver with Master: 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:31.907789 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14349
I20260929 03:27:31.907897 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:31.908051 14136 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:49239
I20260929 03:27:31.908078 6610 external_mini_cluster.cc:1428] Setting key 783a7797bde8e742d7191c7700724162
I20260929 03:27:31.913436 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.907942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:47503 (local address 127.6.116.190:34103)
0929 03:27:31.907979 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:31.907980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:31.908090 (+ 110us) server_negotiation.cc:407] Connection header received
0929 03:27:31.908168 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:31.908169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:31.908178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:31.908221 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:31.908565 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.909139 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.909771 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.909968 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.910046 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.911108 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:31.911109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:31.911110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:31.911116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:31.912325 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.912735 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.912736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.912736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.912792 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:31.913019 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:31.913019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:31.913019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:31.913141 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:31.913187 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:31.913339 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 03:27:31.913386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:27:31.913573 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:31.907893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.130:47503)
0929 03:27:31.908024 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:31.908033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:31.908102 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:31.908234 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:31.908235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:31.908431 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:31.908492 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.908494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.909222 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.909222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.909688 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:31.909689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:31.910049 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:31.910050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:31.910118 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:31.910621 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:31.910627 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:31.911002 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:31.912403 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.912404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.912404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.912637 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.912865 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:31.912866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:31.912866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:31.912933 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:31.913202 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:31.913203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:31.913248 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 03:27:31.913324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:31.913426 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
WARNING: no policy specified for oryx/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
I20260929 03:27:31.933874 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34103
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:32.009788 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:32.009961 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:32.009984 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:32.011006 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:32.011054 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:32.011075 14489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 03:27:32.011132 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:32.012521 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.116.190:34103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:32.012763 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:32.012970 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:32.017439 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:32.017477 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:32.017488 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:32.017634 14489 server_base.cc:1034] running on GCE node
I20260929 03:27:32.017874 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:32.018087 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:32.019238 14489 hybrid_clock.cc:648] HybridClock initialized: now 1790652452019236 us; error 38 us; skew 500 ppm
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:32.021509 14489 init.cc:377] Logged in from keytab as oryx/127.6.116.131@KRBTEST.COM (short username oryx)
I20260929 03:27:32.021941 14489 webserver.cc:460] Webserver started at http://127.6.116.131:37487/ using document root <none> and password file <none>
I20260929 03:27:32.022154 14489 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:32.022195 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:32.022326 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:32.023281 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "afd2385ad7d44521955952b003c0fe0b"
format_stamp: "Formatted at 2026-09-29 03:27:32 on dist-test-slave-z2zg"
server_key: "b9c30a89dcdbf766fff217411727c7ad"
server_key_iv: "ae13356a014e3e1294cd48adf27e19c7"
server_key_version: "encryptionkey@0"
I20260929 03:27:32.023583 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "afd2385ad7d44521955952b003c0fe0b"
format_stamp: "Formatted at 2026-09-29 03:27:32 on dist-test-slave-z2zg"
server_key: "b9c30a89dcdbf766fff217411727c7ad"
server_key_iv: "ae13356a014e3e1294cd48adf27e19c7"
server_key_version: "encryptionkey@0"
I20260929 03:27:32.024768 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:32.025437 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.025604 14489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:32.025686 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "afd2385ad7d44521955952b003c0fe0b"
format_stamp: "Formatted at 2026-09-29 03:27:32 on dist-test-slave-z2zg"
server_key: "b9c30a89dcdbf766fff217411727c7ad"
server_key_iv: "ae13356a014e3e1294cd48adf27e19c7"
server_key_version: "encryptionkey@0"
I20260929 03:27:32.025753 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:32.038829 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:32.040493 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:32.040628 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:32.041122 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:32.041159 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.041199 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:32.041217 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:32.048306 14489 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:45585
I20260929 03:27:32.048358 14619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:45585 every 8 connection(s)
I20260929 03:27:32.048671 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:32.050060 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.041457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:50913 (local address 127.6.116.190:34103)
0929 03:27:32.041505 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.041507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.041697 (+ 190us) server_negotiation.cc:407] Connection header received
0929 03:27:32.042149 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.042151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.042163 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.042225 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.042821 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.043480 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.044401 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.044599 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.044683 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.047340 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:32.047342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:32.047342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:32.047349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:32.048970 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.049332 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.049333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.049333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.049403 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.049639 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.049640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.049640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.049733 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:32.049779 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.049942 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.050002 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 03:27:32.050192 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.041441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.131:50913)
0929 03:27:32.041585 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.041623 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.042033 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.042286 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.042290 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.042571 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:32.042737 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.042741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.043576 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.043577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.044315 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.044317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.044692 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.044693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.044831 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.045334 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:32.045342 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:32.047234 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:32.049046 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.049049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.049051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.049270 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.049483 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.049483 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.049484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.049587 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.049771 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:32.049774 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:32.049863 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.049930 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.050028 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260929 03:27:32.050572 14620 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34103
I20260929 03:27:32.050654 14620 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:32.050819 14620 heartbeater.cc:511] Master 127.6.116.190:34103 requested a full tablet report, sending...
I20260929 03:27:32.051174 14137 ts_manager.cc:194] Registered new tserver with Master: afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585)
I20260929 03:27:32.051644 14137 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:50913
I20260929 03:27:32.056382 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.051517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39315 (local address 127.6.116.190:34103)
0929 03:27:32.051798 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.051800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.051807 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:32.051823 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.051823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.051832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.051870 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.052186 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.052617 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.053266 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.053455 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.053529 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.054283 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:32.054285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:32.054285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:32.054291 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:32.055362 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.055681 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.055682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.055683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.055737 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.055919 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.055920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.055920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.056019 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:32.056084 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.056237 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.056300 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260929 03:27:32.056397 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.051436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.6.116.131:39315)
0929 03:27:32.051569 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.051579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.051635 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.051887 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.051887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.052064 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:32.052115 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.052116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.052698 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.052699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.053180 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.053181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.053530 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.053531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.053592 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.053934 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:32.053938 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:32.054206 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:32.055448 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.055449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.055449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.055614 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.055798 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.055799 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.055799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.055853 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.056090 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:32.056091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:32.056131 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.056202 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.056296 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":92,"threads_started":1}
I20260929 03:27:32.057771 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14489
I20260929 03:27:32.057901 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:32.058084 6610 external_mini_cluster.cc:1428] Setting key 93e920a3f6f1dd4cd5d83d6b3d0ded87
I20260929 03:27:32.058837 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:27:32.077704 14206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:32.073143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54150 (local address 127.6.116.190:34103)
0929 03:27:32.073184 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.073185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.073192 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:32.073240 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.073240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.073247 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.073291 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.073771 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.074208 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.074843 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.075006 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.075103 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.077659 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54150: BlockingRecv error: recv got EOF from 127.0.0.1:54150 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:32.084362 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.078370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.6.116.190:34103)
0929 03:27:32.078410 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.078411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.078505 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:27:32.078540 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.078540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.078547 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.078590 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.078989 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.079457 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.080063 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.080231 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.080277 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.082062 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:32.082064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:32.082064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:32.082070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:32.083344 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.083673 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.083674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.083674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.083726 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.083910 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.083911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.083911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.084004 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:32.084102 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.084198 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.084266 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:27:32.085644 14137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:32.087128 14137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:32.093456 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.6.116.131:45585)
0929 03:27:32.089970 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.089974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.089994 (+ 20us) server_negotiation.cc:407] Connection header received
0929 03:27:32.090023 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.090025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.090038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.090123 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.090240 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091283 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.092514 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093026 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093110 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093180 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.093331 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.093363 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":77,"threads_started":1}
I20260929 03:27:32.093446 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.0.0.1:53240)
0929 03:27:32.089815 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.089823 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.089860 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.090125 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.090126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.090129 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.090169 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.090171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091369 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.091370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.092429 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.092431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093110 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.093187 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093201 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.093292 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.093348 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":73,"threads_started":1}
I20260929 03:27:32.093652 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.6.116.129:41327)
0929 03:27:32.089717 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.089722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.089732 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:32.089798 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.089800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.089816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.089903 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.090192 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091430 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.092320 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093047 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093117 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093187 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.093487 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.093534 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260929 03:27:32.093767 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41327 (local address 127.0.0.1:59760)
0929 03:27:32.089643 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.089652 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.089715 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.090071 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.090072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.090073 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.090114 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.090115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091501 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.091502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.092231 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.092233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093223 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.093292 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093296 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.093342 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.093379 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":76,"threads_started":1}
I20260929 03:27:32.094045 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:37211 (local address 127.0.0.1:44272)
0929 03:27:32.089485 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.089519 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.089732 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.089915 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.089918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.089920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.089989 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.089991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091111 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.091111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.092478 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.092480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093426 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.093495 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093498 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.093940 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.093983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":29,"thread_start_us":60,"threads_started":1}
I20260929 03:27:32.094377 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.089421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.6.116.130:37211)
0929 03:27:32.089580 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.089585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.089736 (+ 151us) server_negotiation.cc:407] Connection header received
0929 03:27:32.089799 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.089801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.089816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.089904 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.090058 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.091004 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.092558 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.093259 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.093331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.093399 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.094249 (+ 850us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.094290 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260929 03:27:32.094352 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 246a2c3588ae485b8967d4579529280f (DEFAULT_TABLE table=test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.094468 14274 tablet_service.cc:1467] Processing CreateTablet for tablet 246a2c3588ae485b8967d4579529280f (DEFAULT_TABLE table=test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.094677 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a2c3588ae485b8967d4579529280f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.094799 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a2c3588ae485b8967d4579529280f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.094909 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 246a2c3588ae485b8967d4579529280f (DEFAULT_TABLE table=test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.095232 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 246a2c3588ae485b8967d4579529280f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.097731 14648 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Bootstrap starting.
I20260929 03:27:32.098150 14649 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Bootstrap starting.
I20260929 03:27:32.098591 14648 tablet_bootstrap.cc:654] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.098685 14650 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Bootstrap starting.
I20260929 03:27:32.098721 14649 tablet_bootstrap.cc:654] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.099035 14649 log.cc:824] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:32.099098 14648 log.cc:824] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:32.099220 14650 tablet_bootstrap.cc:654] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.099553 14650 log.cc:824] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:32.099776 14648 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: No bootstrap required, opened a new log
I20260929 03:27:32.099776 14649 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: No bootstrap required, opened a new log
I20260929 03:27:32.099843 14648 ts_tablet_manager.cc:1397] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:32.099843 14649 ts_tablet_manager.cc:1397] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:32.100162 14650 tablet_bootstrap.cc:492] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: No bootstrap required, opened a new log
I20260929 03:27:32.100234 14650 ts_tablet_manager.cc:1397] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:32.101696 14650 raft_consensus.cc:348] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.101831 14650 raft_consensus.cc:374] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.101872 14650 raft_consensus.cc:729] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.101783 14648 raft_consensus.cc:348] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.101920 14648 raft_consensus.cc:374] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.101961 14648 raft_consensus.cc:729] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.101979 14650 consensus_queue.cc:260] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.101902 14649 raft_consensus.cc:348] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.102059 14649 raft_consensus.cc:374] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.102072 14648 consensus_queue.cc:260] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.102100 14649 raft_consensus.cc:729] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.102222 14650 ts_tablet_manager.cc:1428] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:32.102226 14649 consensus_queue.cc:260] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.102303 14648 ts_tablet_manager.cc:1428] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:27:32.102326 14480 heartbeater.cc:503] Master 127.6.116.190:34103 was elected leader, sending a full tablet report...
I20260929 03:27:32.102427 14340 heartbeater.cc:503] Master 127.6.116.190:34103 was elected leader, sending a full tablet report...
I20260929 03:27:32.102447 14649 ts_tablet_manager.cc:1428] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:32.102553 14620 heartbeater.cc:503] Master 127.6.116.190:34103 was elected leader, sending a full tablet report...
W20260929 03:27:32.154930 14481 tablet.cc:2369] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:32.206947 14655 raft_consensus.cc:482] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:32.207111 14655 raft_consensus.cc:504] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.207468 14655 leader_election.cc:288] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211), afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585)
I20260929 03:27:32.210062 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.207716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:37639 (local address 127.6.116.131:45585)
0929 03:27:32.207770 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.207771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.207780 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:32.207807 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.207808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.207817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.207865 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.208022 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.208485 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209397 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.209742 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209819 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.209827 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.209969 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.210001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:27:32.210081 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.207638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.6.116.129:37639)
0929 03:27:32.207701 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.207712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.207752 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.207882 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.207883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.207884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.207954 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.207955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.208572 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.208573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.209303 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.209304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.209815 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.209901 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.209903 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.209964 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.209999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 03:27:32.210172 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.207692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34097 (local address 127.6.116.130:37211)
0929 03:27:32.207755 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.207757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.207897 (+ 140us) server_negotiation.cc:407] Connection header received
0929 03:27:32.207971 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.207972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.207985 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.208041 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.208190 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.208630 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209518 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.209854 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209931 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.209937 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.210078 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.210117 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:27:32.210186 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.207647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:37211 (local address 127.6.116.129:34097)
0929 03:27:32.207818 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.207826 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.207908 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.208045 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.208046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.208047 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.208119 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.208120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.208709 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.208711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.209431 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.209432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.209931 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.209932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.209999 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.210002 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.210063 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.210101 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":11,"thread_start_us":96,"threads_started":1}
I20260929 03:27:32.210383 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a2c3588ae485b8967d4579529280f" candidate_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd2385ad7d44521955952b003c0fe0b" is_pre_election: true
I20260929 03:27:32.210491 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a2c3588ae485b8967d4579529280f" candidate_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" is_pre_election: true
I20260929 03:27:32.210528 14574 raft_consensus.cc:2454] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e18e717bdd469a9c2b2b7babb0e549 in term 0.
I20260929 03:27:32.210613 14434 raft_consensus.cc:2454] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e18e717bdd469a9c2b2b7babb0e549 in term 0.
I20260929 03:27:32.210749 14226 leader_election.cc:302] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd2385ad7d44521955952b003c0fe0b, e5e18e717bdd469a9c2b2b7babb0e549; no voters:
I20260929 03:27:32.210872 14655 raft_consensus.cc:2781] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:32.210927 14655 raft_consensus.cc:482] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:32.210947 14655 raft_consensus.cc:3037] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.211817 14655 raft_consensus.cc:504] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.211938 14655 leader_election.cc:288] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [CANDIDATE]: Term 1 election: Requested vote from peers 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211), afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585)
I20260929 03:27:32.212083 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a2c3588ae485b8967d4579529280f" candidate_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
I20260929 03:27:32.212100 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246a2c3588ae485b8967d4579529280f" candidate_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd2385ad7d44521955952b003c0fe0b"
I20260929 03:27:32.212157 14434 raft_consensus.cc:3037] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.212168 14574 raft_consensus.cc:3037] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.212966 14574 raft_consensus.cc:2454] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e18e717bdd469a9c2b2b7babb0e549 in term 1.
I20260929 03:27:32.212966 14434 raft_consensus.cc:2454] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e18e717bdd469a9c2b2b7babb0e549 in term 1.
I20260929 03:27:32.213141 14225 leader_election.cc:302] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, e5e18e717bdd469a9c2b2b7babb0e549; no voters:
I20260929 03:27:32.213274 14655 raft_consensus.cc:2781] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:32.213425 14655 raft_consensus.cc:686] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 LEADER]: Becoming Leader. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Running, Role: LEADER
I20260929 03:27:32.213516 14655 consensus_queue.cc:237] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.214277 14137 catalog_manager.cc:5495] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:32.236099 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.234138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.6.116.129:41327)
0929 03:27:32.234187 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.234189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.234195 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:32.234217 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.234217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.234230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.234282 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:32.234375 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.234773 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.235494 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.235672 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.235716 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.235837 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.235913 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.235968 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.236003 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.236017 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:32.237512 14434 raft_consensus.cc:1261] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Refusing update from remote peer e5e18e717bdd469a9c2b2b7babb0e549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.237521 14574 raft_consensus.cc:1261] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Refusing update from remote peer e5e18e717bdd469a9c2b2b7babb0e549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.237805 14658 consensus_queue.cc:1035] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.237905 14655 consensus_queue.cc:1035] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.239816 14661 mvcc.cc:205] Tried to move back new op lower bound from 7334512444362629120 to 7334512444267134976. Current Snapshot: MvccSnapshot[applied={T|T < 7334512444362629120}]
I20260929 03:27:32.240592 14662 mvcc.cc:205] Tried to move back new op lower bound from 7334512444362629120 to 7334512444267134976. Current Snapshot: MvccSnapshot[applied={T|T < 7334512444362629120}]
I20260929 03:27:32.241695 14137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:32.241781 14137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:32.242406 14137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Sending DeleteTablet for 3 replicas of tablet 246a2c3588ae485b8967d4579529280f
I20260929 03:27:32.242735 14274 tablet_service.cc:1514] Processing DeleteTablet for tablet 246a2c3588ae485b8967d4579529280f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:59760
I20260929 03:27:32.242748 14554 tablet_service.cc:1514] Processing DeleteTablet for tablet 246a2c3588ae485b8967d4579529280f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53240
I20260929 03:27:32.242955 14671 tablet_replica.cc:330] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: stopping tablet replica
I20260929 03:27:32.242939 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 246a2c3588ae485b8967d4579529280f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260929 03:27:32.243127 14671 raft_consensus.cc:2229] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:32.243260 14671 raft_consensus.cc:2258] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.243325 14673 tablet_replica.cc:330] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: stopping tablet replica
I20260929 03:27:32.243299 14137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 03:27:32.243410 14673 raft_consensus.cc:2229] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 03:27:32.243439 14137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:32.243731 14672 tablet_replica.cc:330] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: stopping tablet replica
I20260929 03:27:32.243870 14672 raft_consensus.cc:2229] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:32.244249 14672 raft_consensus.cc:2258] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.244648 14274 tablet_service.cc:1467] Processing CreateTablet for tablet 1d290df433a2436bbe73111b3d519916 (DEFAULT_TABLE table=test-table [id=e0e264639d6048f8a5db603944c5cea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.244760 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d290df433a2436bbe73111b3d519916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.244838 14672 ts_tablet_manager.cc:1905] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.246127 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 1d290df433a2436bbe73111b3d519916 (DEFAULT_TABLE table=test-table [id=e0e264639d6048f8a5db603944c5cea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.246249 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d290df433a2436bbe73111b3d519916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.246557 14672 ts_tablet_manager.cc:1918] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:32.246658 14672 log.cc:1196] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/246a2c3588ae485b8967d4579529280f
I20260929 03:27:32.246999 14672 ts_tablet_manager.cc:1939] T 246a2c3588ae485b8967d4579529280f P afd2385ad7d44521955952b003c0fe0b: Deleting consensus metadata
I20260929 03:27:32.247664 14650 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Bootstrap starting.
I20260929 03:27:32.247694 14648 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Bootstrap starting.
I20260929 03:27:32.248049 14648 tablet_bootstrap.cc:654] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.248126 14650 tablet_bootstrap.cc:654] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.248521 14122 catalog_manager.cc:4841] TS afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585): tablet 246a2c3588ae485b8967d4579529280f (table test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]) successfully deleted
I20260929 03:27:32.248615 14554 tablet_service.cc:1467] Processing CreateTablet for tablet 1d290df433a2436bbe73111b3d519916 (DEFAULT_TABLE table=test-table [id=e0e264639d6048f8a5db603944c5cea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.248711 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d290df433a2436bbe73111b3d519916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.248728 14671 ts_tablet_manager.cc:1905] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.249841 14649 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Bootstrap starting.
I20260929 03:27:32.249898 14671 ts_tablet_manager.cc:1918] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:32.249950 14671 log.cc:1196] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/246a2c3588ae485b8967d4579529280f
I20260929 03:27:32.250193 14671 ts_tablet_manager.cc:1939] T 246a2c3588ae485b8967d4579529280f P e5e18e717bdd469a9c2b2b7babb0e549: Deleting consensus metadata
I20260929 03:27:32.250221 14649 tablet_bootstrap.cc:654] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.250559 14124 catalog_manager.cc:4841] TS e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327): tablet 246a2c3588ae485b8967d4579529280f (table test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]) successfully deleted
I20260929 03:27:32.250862 14649 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: No bootstrap required, opened a new log
I20260929 03:27:32.250864 14673 raft_consensus.cc:2258] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.250916 14649 ts_tablet_manager.cc:1397] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:32.251080 14649 raft_consensus.cc:348] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.251170 14649 raft_consensus.cc:374] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.251199 14649 raft_consensus.cc:729] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.251235 14649 consensus_queue.cc:260] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.251300 14648 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: No bootstrap required, opened a new log
I20260929 03:27:32.251322 14649 ts_tablet_manager.cc:1428] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.251340 14648 ts_tablet_manager.cc:1397] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 03:27:32.251470 14648 raft_consensus.cc:348] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.251533 14648 raft_consensus.cc:374] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.251554 14648 raft_consensus.cc:729] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.251596 14648 consensus_queue.cc:260] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.251714 14648 ts_tablet_manager.cc:1428] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:32.251726 14673 ts_tablet_manager.cc:1905] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.252396 14650 tablet_bootstrap.cc:492] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: No bootstrap required, opened a new log
I20260929 03:27:32.252449 14650 ts_tablet_manager.cc:1397] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:27:32.252575 14650 raft_consensus.cc:348] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.252640 14650 raft_consensus.cc:374] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.252668 14650 raft_consensus.cc:729] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.252705 14650 consensus_queue.cc:260] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.252780 14650 ts_tablet_manager.cc:1428] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:32.253278 14673 ts_tablet_manager.cc:1918] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:32.253353 14673 log.cc:1196] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/246a2c3588ae485b8967d4579529280f
I20260929 03:27:32.253558 14673 ts_tablet_manager.cc:1939] T 246a2c3588ae485b8967d4579529280f P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting consensus metadata
I20260929 03:27:32.253872 14121 catalog_manager.cc:4841] TS 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211): tablet 246a2c3588ae485b8967d4579529280f (table test-table [id=c5974b62ad3b4e07b9cc43134d9e8783]) successfully deleted
W20260929 03:27:32.299206 14621 tablet.cc:2369] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:32.483115 14656 raft_consensus.cc:482] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:32.483222 14656 raft_consensus.cc:504] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.483583 14656 leader_election.cc:288] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:32.486326 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.483810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:38311 (local address 127.6.116.129:41327)
0929 03:27:32.483871 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.483873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.483969 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:27:32.484000 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.484001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.484012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.484072 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.484255 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.484718 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.485621 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.485956 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.486055 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.486062 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.486232 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.486268 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:27:32.486311 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.483711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:37211 (local address 127.6.116.131:51455)
0929 03:27:32.483771 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.483783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.483825 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.483984 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.483985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.483986 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.484062 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.484064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.484710 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.484711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.485476 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.485478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.486041 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.486042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.486128 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.486131 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.486172 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.486238 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:27:32.486320 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.483823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51455 (local address 127.6.116.130:37211)
0929 03:27:32.483865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.483867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.483876 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:32.483903 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.483904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.483915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.483969 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:32.484136 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.484640 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.485564 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.485958 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.486013 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.486039 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.486197 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.486244 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:32.486431 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.483712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41327 (local address 127.6.116.131:38311)
0929 03:27:32.483885 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.483895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.483958 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.484083 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.484084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.484085 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:32.484189 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.484190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.484791 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.484792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.485502 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.485504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.486086 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.486092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.486137 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.486140 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.486218 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.486251 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":50,"thread_start_us":53,"threads_started":1}
I20260929 03:27:32.486517 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d290df433a2436bbe73111b3d519916" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" is_pre_election: true
I20260929 03:27:32.486598 14434 raft_consensus.cc:2454] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:32.486688 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d290df433a2436bbe73111b3d519916" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" is_pre_election: true
I20260929 03:27:32.486744 14505 leader_election.cc:302] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, afd2385ad7d44521955952b003c0fe0b; no voters:
I20260929 03:27:32.486830 14294 raft_consensus.cc:2454] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:32.486868 14656 raft_consensus.cc:2781] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:32.486927 14656 raft_consensus.cc:482] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:32.486961 14656 raft_consensus.cc:3037] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.487666 14656 raft_consensus.cc:504] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.487783 14656 leader_election.cc:288] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Requested vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:32.487917 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d290df433a2436bbe73111b3d519916" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
I20260929 03:27:32.487920 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d290df433a2436bbe73111b3d519916" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
I20260929 03:27:32.487978 14294 raft_consensus.cc:3037] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.487982 14434 raft_consensus.cc:3037] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.488490 14434 raft_consensus.cc:2454] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:32.488677 14505 leader_election.cc:302] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, afd2385ad7d44521955952b003c0fe0b; no voters:
I20260929 03:27:32.488691 14294 raft_consensus.cc:2454] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:32.488783 14656 raft_consensus.cc:2781] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:32.488922 14656 raft_consensus.cc:686] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 1 LEADER]: Becoming Leader. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Running, Role: LEADER
I20260929 03:27:32.488976 14656 consensus_queue.cc:237] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.489547 14137 catalog_manager.cc:5495] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b reported cstate change: term changed from 0 to 1, leader changed from <none> to afd2385ad7d44521955952b003c0fe0b (127.6.116.131). New cstate: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } health_report { overall_health: HEALTHY } } }
W20260929 03:27:32.496273 14341 tablet.cc:2369] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652451, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.190@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:32.562081 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.555698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.6.116.190:34103)
0929 03:27:32.555758 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.555761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.555858 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:27:32.555949 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.555950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.555964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.556020 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.556433 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.556912 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.557604 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.557770 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.557859 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.559822 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:32.559823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:32.559824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:32.559831 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:32.560975 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.561336 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.561337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.561338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.561391 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.561569 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.561570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.561571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.561682 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:32.561751 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.561893 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.561974 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:27:32.562152 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.555650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34103 (local address 127.0.0.1:54162)
0929 03:27:32.555792 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.555800 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.555873 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.556025 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.556026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.556312 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:32.556353 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.556355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.556989 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.556990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.557504 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.557506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.557858 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.557859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.557915 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.558395 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:32.558399 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:32.559714 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:32.561066 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.561073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.561076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.561266 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.561451 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:32.561452 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:32.561453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:32.561511 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:32.561742 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:32.561748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:32.561791 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.561869 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.562026 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":64,"thread_start_us":77,"threads_started":1}
I20260929 03:27:32.563974 14137 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:54162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 03:27:32.564155 14137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:32.565379 14554 tablet_service.cc:1467] Processing CreateTablet for tablet c2806dda8e8f414aa9b97052265e4596 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d8a7a6aa3c740adbcfae54558b4b2a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:32.565379 14414 tablet_service.cc:1467] Processing CreateTablet for tablet c2806dda8e8f414aa9b97052265e4596 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d8a7a6aa3c740adbcfae54558b4b2a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:32.565392 14274 tablet_service.cc:1467] Processing CreateTablet for tablet c2806dda8e8f414aa9b97052265e4596 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0d8a7a6aa3c740adbcfae54558b4b2a6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 03:27:32.565527 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2806dda8e8f414aa9b97052265e4596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.565526 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2806dda8e8f414aa9b97052265e4596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.565546 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2806dda8e8f414aa9b97052265e4596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.566722 14648 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: Bootstrap starting.
I20260929 03:27:32.566974 14650 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: Bootstrap starting.
I20260929 03:27:32.566974 14649 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: Bootstrap starting.
I20260929 03:27:32.567086 14648 tablet_bootstrap.cc:654] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.567390 14649 tablet_bootstrap.cc:654] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.567390 14650 tablet_bootstrap.cc:654] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.567672 14648 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: No bootstrap required, opened a new log
I20260929 03:27:32.567725 14648 ts_tablet_manager.cc:1397] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:32.567847 14648 raft_consensus.cc:348] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.567922 14648 raft_consensus.cc:374] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.567955 14648 raft_consensus.cc:729] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.567989 14649 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: No bootstrap required, opened a new log
I20260929 03:27:32.568005 14650 tablet_bootstrap.cc:492] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: No bootstrap required, opened a new log
I20260929 03:27:32.567996 14648 consensus_queue.cc:260] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.568046 14650 ts_tablet_manager.cc:1397] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:32.568046 14649 ts_tablet_manager.cc:1397] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:32.568082 14648 ts_tablet_manager.cc:1428] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:32.568190 14650 raft_consensus.cc:348] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.568192 14649 raft_consensus.cc:348] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.568215 14668 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.568260 14650 raft_consensus.cc:374] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.568260 14649 raft_consensus.cc:374] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.568284 14649 raft_consensus.cc:729] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.568284 14650 raft_consensus.cc:729] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.568287 14668 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.568326 14649 consensus_queue.cc:260] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.568326 14650 consensus_queue.cc:260] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.568410 14649 ts_tablet_manager.cc:1428] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:32.568410 14650 ts_tablet_manager.cc:1428] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.568558 14654 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.568630 14654 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.568745 14656 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.568837 14656 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.774144 14656 raft_consensus.cc:482] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:32.774230 14656 raft_consensus.cc:504] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.774400 14656 leader_election.cc:288] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:32.774605 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2806dda8e8f414aa9b97052265e4596" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" is_pre_election: true
I20260929 03:27:32.774614 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2806dda8e8f414aa9b97052265e4596" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" is_pre_election: true
I20260929 03:27:32.774699 14434 raft_consensus.cc:2454] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:32.774700 14294 raft_consensus.cc:2454] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:32.774842 14505 leader_election.cc:302] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, afd2385ad7d44521955952b003c0fe0b; no voters:
I20260929 03:27:32.774938 14656 raft_consensus.cc:2781] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:32.774986 14656 raft_consensus.cc:482] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:32.775000 14656 raft_consensus.cc:3037] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.775660 14656 raft_consensus.cc:504] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.775780 14656 leader_election.cc:288] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Requested vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:32.775933 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2806dda8e8f414aa9b97052265e4596" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
I20260929 03:27:32.776005 14294 raft_consensus.cc:3037] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.776018 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2806dda8e8f414aa9b97052265e4596" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
I20260929 03:27:32.776098 14434 raft_consensus.cc:3037] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:32.776594 14294 raft_consensus.cc:2454] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:32.776715 14434 raft_consensus.cc:2454] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:32.776751 14508 leader_election.cc:302] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd2385ad7d44521955952b003c0fe0b, e5e18e717bdd469a9c2b2b7babb0e549; no voters:
I20260929 03:27:32.776837 14656 raft_consensus.cc:2781] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:32.776894 14656 raft_consensus.cc:686] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [term 1 LEADER]: Becoming Leader. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Running, Role: LEADER
I20260929 03:27:32.776937 14656 consensus_queue.cc:237] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.777240 14678 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: TxnStatusTablet state changed. Reason: New leader afd2385ad7d44521955952b003c0fe0b. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.777335 14678 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:32.777474 14690 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 03:27:32.777629 14690 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 03:27:32.777549 14136 catalog_manager.cc:5495] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b reported cstate change: term changed from 0 to 1, leader changed from <none> to afd2385ad7d44521955952b003c0fe0b (127.6.116.131). New cstate: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } health_report { overall_health: HEALTHY } } }
I20260929 03:27:32.794011 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.791483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.6.116.131:45585)
0929 03:27:32.791582 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.791584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.791593 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:32.791649 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.791650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.791666 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.791750 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:32.791863 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.792475 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.793217 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.793422 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.793492 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.793681 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.793771 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.793849 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.793904 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.793922 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":47,"threads_started":1}
I20260929 03:27:32.794027 14685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.791432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.0.0.1:53242)
0929 03:27:32.791505 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.791517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.791566 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.791755 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.791756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.791757 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:32.791807 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.791809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.792561 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.792562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.793136 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.793138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.793501 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.793502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.793566 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.793584 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.793835 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.793843 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.793898 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.793945 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:27:32.795069 14434 raft_consensus.cc:1261] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.795151 14294 raft_consensus.cc:1261] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.795461 14678 consensus_queue.cc:1035] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.795588 14656 consensus_queue.cc:1035] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.796155 14654 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: TxnStatusTablet state changed. Reason: New leader afd2385ad7d44521955952b003c0fe0b. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.796170 14668 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: TxnStatusTablet state changed. Reason: New leader afd2385ad7d44521955952b003c0fe0b. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.796239 14668 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.796239 14654 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.797042 14656 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.797127 14656 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:32.797190 14668 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.797271 14668 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P e5e18e717bdd469a9c2b2b7babb0e549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.797514 14678 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.797588 14678 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P afd2385ad7d44521955952b003c0fe0b: This TxnStatusTablet replica's current role is: LEADER
I20260929 03:27:32.797919 14654 tablet_replica.cc:439] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } } }
I20260929 03:27:32.797992 14654 tablet_replica.cc:442] T c2806dda8e8f414aa9b97052265e4596 P 0e53e6638536436f9ca6aae65b2bcf4e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 03:27:32.798163 14696 mvcc.cc:205] Tried to move back new op lower bound from 7334512446646894592 to 7334512446574882816. Current Snapshot: MvccSnapshot[applied={T|T < 7334512446646894592}]
I20260929 03:27:32.805315 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.799124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.6.116.131:45585)
0929 03:27:32.799162 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.799164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.803216 (+ 4052us) server_negotiation.cc:407] Connection header received
0929 03:27:32.803281 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.803282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.803292 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.803335 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:32.803485 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.803909 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.804646 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.804834 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.804902 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.805072 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.805106 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.805157 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.805227 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.805244 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:32.809999 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.807501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39725 (local address 127.6.116.131:45585)
0929 03:27:32.807536 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.807538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.807657 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:27:32.807713 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.807714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.807720 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.807763 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:32.807892 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.808300 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.809400 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.809571 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.809621 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.809808 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.809828 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.809865 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.809934 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.809947 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:32.810313 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.807455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.6.116.131:39725)
0929 03:27:32.807588 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:32.807598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:32.807659 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:32.807782 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:32.807782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:32.807784 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:32.807838 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.807839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.808364 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.808365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.809325 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:32.809328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.809643 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.809644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:32.809724 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.809736 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:32.809882 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:32.809884 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:32.809933 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:27:32.809971 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 03:27:32.812286 14434 raft_consensus.cc:1261] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.812316 14294 raft_consensus.cc:1261] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:32.812480 14656 consensus_queue.cc:1035] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.812559 14678 consensus_queue.cc:1035] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:32.823091 14136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:32.823174 14136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:32.823760 14136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Sending DeleteTablet for 3 replicas of tablet 1d290df433a2436bbe73111b3d519916
I20260929 03:27:32.823975 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d290df433a2436bbe73111b3d519916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260929 03:27:32.824040 14553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d290df433a2436bbe73111b3d519916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53240
I20260929 03:27:32.824146 14712 tablet_replica.cc:330] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: stopping tablet replica
I20260929 03:27:32.824189 14712 raft_consensus.cc:2229] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:32.824222 14712 raft_consensus.cc:2258] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.824438 14712 ts_tablet_manager.cc:1905] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.826009 14712 ts_tablet_manager.cc:1918] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:32.826081 14712 log.cc:1196] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/1d290df433a2436bbe73111b3d519916
I20260929 03:27:32.826320 14712 ts_tablet_manager.cc:1939] T 1d290df433a2436bbe73111b3d519916 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting consensus metadata
I20260929 03:27:32.826627 14121 catalog_manager.cc:4841] TS 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211): tablet 1d290df433a2436bbe73111b3d519916 (table test-table [id=e0e264639d6048f8a5db603944c5cea7]) successfully deleted
I20260929 03:27:32.826769 14713 tablet_replica.cc:330] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: stopping tablet replica
I20260929 03:27:32.826823 14713 raft_consensus.cc:2229] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:32.826848 14274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d290df433a2436bbe73111b3d519916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:32 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:59760
I20260929 03:27:32.826889 14713 raft_consensus.cc:2258] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.827091 14715 tablet_replica.cc:330] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: stopping tablet replica
I20260929 03:27:32.827113 14713 ts_tablet_manager.cc:1905] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.827150 14715 raft_consensus.cc:2229] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:32.827193 14715 raft_consensus.cc:2258] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:32.827412 14715 ts_tablet_manager.cc:1905] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:32.828635 14713 ts_tablet_manager.cc:1918] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:32.828694 14713 log.cc:1196] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/1d290df433a2436bbe73111b3d519916
I20260929 03:27:32.828904 14713 ts_tablet_manager.cc:1939] T 1d290df433a2436bbe73111b3d519916 P afd2385ad7d44521955952b003c0fe0b: Deleting consensus metadata
I20260929 03:27:32.828909 14715 ts_tablet_manager.cc:1918] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:32.828979 14715 log.cc:1196] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/1d290df433a2436bbe73111b3d519916
I20260929 03:27:32.829164 14122 catalog_manager.cc:4841] TS afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585): tablet 1d290df433a2436bbe73111b3d519916 (table test-table [id=e0e264639d6048f8a5db603944c5cea7]) successfully deleted
I20260929 03:27:32.829191 14715 ts_tablet_manager.cc:1939] T 1d290df433a2436bbe73111b3d519916 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting consensus metadata
I20260929 03:27:32.829558 14124 catalog_manager.cc:4841] TS e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327): tablet 1d290df433a2436bbe73111b3d519916 (table test-table [id=e0e264639d6048f8a5db603944c5cea7]) successfully deleted
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:27:32.850216 14206 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:32.844172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.6.116.190:34103)
0929 03:27:32.844227 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.844229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.844348 (+ 119us) server_negotiation.cc:407] Connection header received
0929 03:27:32.844414 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.844415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.844423 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.844470 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.845192 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.845675 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.846734 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.846949 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.847016 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.850133 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54174: BlockingRecv error: recv got EOF from 127.0.0.1:54174 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 29 03:27:32 dist-test-slave-z2zg krb5kdc[14092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM
I20260929 03:27:32.861713 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:32.851010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.6.116.190:34103)
0929 03:27:32.851070 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:32.851072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:32.854856 (+ 3784us) server_negotiation.cc:407] Connection header received
0929 03:27:32.854952 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:32.854953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:32.854964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:32.855045 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:32.855566 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.856074 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.856810 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:32.857019 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:32.857107 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:32.859310 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:32.859312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:32.859312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:32.859320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:32.860631 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.860959 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.860960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.860960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.861021 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:32.861229 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:32.861230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:32.861231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:32.861334 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:32.861434 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:32.861530 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:27:32.861605 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:32.863296 14136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:32.863463 14136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:32.864876 14274 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 (DEFAULT_TABLE table=test-table [id=3737ba67009945d7b26f7a57c5c63227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.864931 14553 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 (DEFAULT_TABLE table=test-table [id=3737ba67009945d7b26f7a57c5c63227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.865007 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2d01db98d54d709d456af79ac0b6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.864991 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 (DEFAULT_TABLE table=test-table [id=3737ba67009945d7b26f7a57c5c63227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:32.865031 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2d01db98d54d709d456af79ac0b6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.865126 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2d01db98d54d709d456af79ac0b6b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:32.866359 14648 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Bootstrap starting.
I20260929 03:27:32.866519 14650 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Bootstrap starting.
I20260929 03:27:32.866549 14649 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Bootstrap starting.
I20260929 03:27:32.866700 14648 tablet_bootstrap.cc:654] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.866919 14650 tablet_bootstrap.cc:654] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.866991 14649 tablet_bootstrap.cc:654] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:32.867385 14648 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: No bootstrap required, opened a new log
I20260929 03:27:32.867424 14648 ts_tablet_manager.cc:1397] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:32.867506 14648 raft_consensus.cc:348] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.867606 14648 raft_consensus.cc:374] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.867635 14648 raft_consensus.cc:729] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.867678 14648 consensus_queue.cc:260] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.867765 14648 ts_tablet_manager.cc:1428] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.868081 14649 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: No bootstrap required, opened a new log
I20260929 03:27:32.868132 14649 ts_tablet_manager.cc:1397] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:32.868275 14649 raft_consensus.cc:348] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.868355 14649 raft_consensus.cc:374] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.868381 14649 raft_consensus.cc:729] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.868415 14649 consensus_queue.cc:260] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.868436 14650 tablet_bootstrap.cc:492] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: No bootstrap required, opened a new log
I20260929 03:27:32.868489 14650 ts_tablet_manager.cc:1397] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:27:32.868515 14649 ts_tablet_manager.cc:1428] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:32.868645 14650 raft_consensus.cc:348] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.868711 14650 raft_consensus.cc:374] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:32.868736 14650 raft_consensus.cc:729] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Initialized, Role: FOLLOWER
I20260929 03:27:32.868782 14650 consensus_queue.cc:260] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:32.868888 14650 ts_tablet_manager.cc:1428] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:33.249717 14678 raft_consensus.cc:482] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:33.249801 14678 raft_consensus.cc:504] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.249985 14678 leader_election.cc:288] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:33.250190 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2d01db98d54d709d456af79ac0b6b5" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" is_pre_election: true
I20260929 03:27:33.250223 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2d01db98d54d709d456af79ac0b6b5" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" is_pre_election: true
I20260929 03:27:33.250289 14294 raft_consensus.cc:2454] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:33.250298 14434 raft_consensus.cc:2454] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 0.
I20260929 03:27:33.250445 14508 leader_election.cc:302] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd2385ad7d44521955952b003c0fe0b, e5e18e717bdd469a9c2b2b7babb0e549; no voters:
I20260929 03:27:33.250595 14678 raft_consensus.cc:2781] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:33.250638 14678 raft_consensus.cc:482] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:33.250662 14678 raft_consensus.cc:3037] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.251466 14678 raft_consensus.cc:504] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.251605 14678 leader_election.cc:288] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Requested vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211)
I20260929 03:27:33.251744 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2d01db98d54d709d456af79ac0b6b5" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
I20260929 03:27:33.251772 14434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2d01db98d54d709d456af79ac0b6b5" candidate_uuid: "afd2385ad7d44521955952b003c0fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53e6638536436f9ca6aae65b2bcf4e"
I20260929 03:27:33.251837 14294 raft_consensus.cc:3037] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.251843 14434 raft_consensus.cc:3037] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.252429 14434 raft_consensus.cc:2454] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:33.252431 14294 raft_consensus.cc:2454] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd2385ad7d44521955952b003c0fe0b in term 1.
I20260929 03:27:33.252599 14508 leader_election.cc:302] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afd2385ad7d44521955952b003c0fe0b, e5e18e717bdd469a9c2b2b7babb0e549; no voters:
I20260929 03:27:33.252732 14678 raft_consensus.cc:2781] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:33.252804 14678 raft_consensus.cc:686] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 1 LEADER]: Becoming Leader. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Running, Role: LEADER
I20260929 03:27:33.252852 14678 consensus_queue.cc:237] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.253332 14135 catalog_manager.cc:5495] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b reported cstate change: term changed from 0 to 1, leader changed from <none> to afd2385ad7d44521955952b003c0fe0b (127.6.116.131). New cstate: current_term: 1 leader_uuid: "afd2385ad7d44521955952b003c0fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } health_report { overall_health: HEALTHY } } }
I20260929 03:27:33.309167 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.307034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.6.116.131:45585)
0929 03:27:33.307101 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.307103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.307135 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:27:33.307203 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.307204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.307215 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.307270 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:33.307398 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.307916 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.308627 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.308792 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.308842 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.308950 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.308973 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.309016 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.309071 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.309088 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:27:33.309819 14434 raft_consensus.cc:1261] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:33.309896 14294 raft_consensus.cc:1261] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Refusing update from remote peer afd2385ad7d44521955952b003c0fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:33.310053 14678 consensus_queue.cc:1035] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:33.310153 14710 consensus_queue.cc:1035] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:33.312711 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:33.312810 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:33.313287 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Sending DeleteTablet for 3 replicas of tablet 1e2d01db98d54d709d456af79ac0b6b5
I20260929 03:27:33.313519 14553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53240
I20260929 03:27:33.313513 14274 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:59760
I20260929 03:27:33.313537 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2d01db98d54d709d456af79ac0b6b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260929 03:27:33.313601 14713 tablet_replica.cc:330] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: stopping tablet replica
I20260929 03:27:33.313601 14715 tablet_replica.cc:330] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: stopping tablet replica
I20260929 03:27:33.313616 14712 tablet_replica.cc:330] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: stopping tablet replica
I20260929 03:27:33.313652 14715 raft_consensus.cc:2229] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:33.313650 14713 raft_consensus.cc:2229] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:33.313661 14712 raft_consensus.cc:2229] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:33.313694 14715 raft_consensus.cc:2258] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.313696 14712 raft_consensus.cc:2258] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.313707 14713 raft_consensus.cc:2258] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.313979 14715 ts_tablet_manager.cc:1905] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.313984 14712 ts_tablet_manager.cc:1905] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.314177 14135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:33.314311 14135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:33.315446 14553 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97c66ea90b4c279675588114374730 (DEFAULT_TABLE table=test-table [id=1c99e0dd1a6542d6bd685c068c56b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:33.315575 14553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97c66ea90b4c279675588114374730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:33.315611 14715 ts_tablet_manager.cc:1918] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:33.315627 14712 ts_tablet_manager.cc:1918] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:33.315662 14715 log.cc:1196] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/1e2d01db98d54d709d456af79ac0b6b5
I20260929 03:27:33.315646 14414 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97c66ea90b4c279675588114374730 (DEFAULT_TABLE table=test-table [id=1c99e0dd1a6542d6bd685c068c56b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:33.315675 14712 log.cc:1196] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/1e2d01db98d54d709d456af79ac0b6b5
I20260929 03:27:33.315740 14414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97c66ea90b4c279675588114374730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:33.315884 14715 ts_tablet_manager.cc:1939] T 1e2d01db98d54d709d456af79ac0b6b5 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting consensus metadata
I20260929 03:27:33.315891 14712 ts_tablet_manager.cc:1939] T 1e2d01db98d54d709d456af79ac0b6b5 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting consensus metadata
I20260929 03:27:33.316017 14274 tablet_service.cc:1467] Processing CreateTablet for tablet 7f97c66ea90b4c279675588114374730 (DEFAULT_TABLE table=test-table [id=1c99e0dd1a6542d6bd685c068c56b853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:33.316087 14274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f97c66ea90b4c279675588114374730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:33.316812 14649 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Bootstrap starting.
I20260929 03:27:33.316864 14650 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Bootstrap starting.
I20260929 03:27:33.317098 14649 tablet_bootstrap.cc:654] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:33.317189 14650 tablet_bootstrap.cc:654] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:33.317502 14648 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Bootstrap starting.
I20260929 03:27:33.317498 14124 catalog_manager.cc:4841] TS e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327): tablet 1e2d01db98d54d709d456af79ac0b6b5 (table test-table [id=3737ba67009945d7b26f7a57c5c63227]) successfully deleted
I20260929 03:27:33.317620 14649 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: No bootstrap required, opened a new log
I20260929 03:27:33.317689 14649 ts_tablet_manager.cc:1397] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:27:33.317812 14648 tablet_bootstrap.cc:654] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:33.317808 14649 raft_consensus.cc:348] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.317860 14649 raft_consensus.cc:374] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:33.317873 14649 raft_consensus.cc:729] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd2385ad7d44521955952b003c0fe0b, State: Initialized, Role: FOLLOWER
I20260929 03:27:33.317932 14649 consensus_queue.cc:260] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.318015 14649 ts_tablet_manager.cc:1428] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:33.318149 14713 ts_tablet_manager.cc:1905] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.319164 14121 catalog_manager.cc:4841] TS 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211): tablet 1e2d01db98d54d709d456af79ac0b6b5 (table test-table [id=3737ba67009945d7b26f7a57c5c63227]) successfully deleted
I20260929 03:27:33.319365 14713 ts_tablet_manager.cc:1918] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:33.319427 14713 log.cc:1196] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/1e2d01db98d54d709d456af79ac0b6b5
I20260929 03:27:33.319444 14650 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: No bootstrap required, opened a new log
I20260929 03:27:33.319496 14650 ts_tablet_manager.cc:1397] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 03:27:33.319639 14713 ts_tablet_manager.cc:1939] T 1e2d01db98d54d709d456af79ac0b6b5 P afd2385ad7d44521955952b003c0fe0b: Deleting consensus metadata
I20260929 03:27:33.319680 14650 raft_consensus.cc:348] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.319773 14650 raft_consensus.cc:374] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:33.319792 14650 raft_consensus.cc:729] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Initialized, Role: FOLLOWER
I20260929 03:27:33.319825 14650 consensus_queue.cc:260] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.319904 14650 ts_tablet_manager.cc:1428] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:33.320020 14122 catalog_manager.cc:4841] TS afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585): tablet 1e2d01db98d54d709d456af79ac0b6b5 (table test-table [id=3737ba67009945d7b26f7a57c5c63227]) successfully deleted
I20260929 03:27:33.320344 14648 tablet_bootstrap.cc:492] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: No bootstrap required, opened a new log
I20260929 03:27:33.320397 14648 ts_tablet_manager.cc:1397] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 03:27:33.320497 14648 raft_consensus.cc:348] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.320567 14648 raft_consensus.cc:374] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:33.320585 14648 raft_consensus.cc:729] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e18e717bdd469a9c2b2b7babb0e549, State: Initialized, Role: FOLLOWER
I20260929 03:27:33.320629 14648 consensus_queue.cc:260] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.320730 14648 ts_tablet_manager.cc:1428] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:33.533499 14654 raft_consensus.cc:482] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:33.533604 14654 raft_consensus.cc:504] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.534129 14654 leader_election.cc:288] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585)
I20260929 03:27:33.537096 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.534322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.6.116.130:37301)
0929 03:27:33.534423 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:33.534436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:33.534505 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:33.534680 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:33.534681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:33.534683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:33.534760 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.534762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.535466 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.535467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.536309 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.536311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.536835 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.536836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.536908 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.536911 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:33.536975 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:27:33.537012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
I20260929 03:27:33.537088 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.534391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:37301 (local address 127.6.116.131:45585)
0929 03:27:33.534440 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.534443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.534499 (+ 56us) server_negotiation.cc:407] Connection header received
0929 03:27:33.534591 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.534592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.534601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.534677 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:33.534839 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.535382 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.536396 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.536745 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.536827 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.536835 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.536979 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.537013 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 03:27:33.537339 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97c66ea90b4c279675588114374730" candidate_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd2385ad7d44521955952b003c0fe0b" is_pre_election: true
I20260929 03:27:33.537420 14574 raft_consensus.cc:2454] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e53e6638536436f9ca6aae65b2bcf4e in term 0.
I20260929 03:27:33.537470 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.534324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:41327 (local address 127.6.116.130:40477)
0929 03:27:33.534480 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:33.534490 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:33.534555 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:33.534754 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:33.534755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:33.534757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:33.534838 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.534839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.535513 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.535514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.536387 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.536390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.537010 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.537011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.537094 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.537097 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:33.537322 (+ 225us) client_negotiation.cc:240] Negotiation successful
0929 03:27:33.537362 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":54,"threads_started":1}
I20260929 03:27:33.537693 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.534499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:40477 (local address 127.6.116.129:41327)
0929 03:27:33.534602 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.534604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.534612 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:33.534641 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.534642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.534660 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.534736 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:33.534916 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.535417 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.536504 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.536879 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.537012 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.537021 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.537573 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.537629 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
I20260929 03:27:33.537667 14366 leader_election.cc:302] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, afd2385ad7d44521955952b003c0fe0b; no voters:
I20260929 03:27:33.537799 14654 raft_consensus.cc:2781] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:33.537834 14654 raft_consensus.cc:482] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:33.537856 14654 raft_consensus.cc:3037] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.537917 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97c66ea90b4c279675588114374730" candidate_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" is_pre_election: true
I20260929 03:27:33.537997 14294 raft_consensus.cc:2454] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e53e6638536436f9ca6aae65b2bcf4e in term 0.
I20260929 03:27:33.538584 14654 raft_consensus.cc:504] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.538715 14654 leader_election.cc:288] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [CANDIDATE]: Term 1 election: Requested vote from peers e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327), afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585)
I20260929 03:27:33.538861 14574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97c66ea90b4c279675588114374730" candidate_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afd2385ad7d44521955952b003c0fe0b"
I20260929 03:27:33.538980 14574 raft_consensus.cc:3037] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.539189 14294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f97c66ea90b4c279675588114374730" candidate_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5e18e717bdd469a9c2b2b7babb0e549"
I20260929 03:27:33.539273 14294 raft_consensus.cc:3037] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:33.539687 14574 raft_consensus.cc:2454] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e53e6638536436f9ca6aae65b2bcf4e in term 1.
I20260929 03:27:33.539853 14366 leader_election.cc:302] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e53e6638536436f9ca6aae65b2bcf4e, afd2385ad7d44521955952b003c0fe0b; no voters:
I20260929 03:27:33.539992 14294 raft_consensus.cc:2454] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e53e6638536436f9ca6aae65b2bcf4e in term 1.
I20260929 03:27:33.540635 14654 raft_consensus.cc:2781] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:33.540783 14654 raft_consensus.cc:686] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 LEADER]: Becoming Leader. State: Replica: 0e53e6638536436f9ca6aae65b2bcf4e, State: Running, Role: LEADER
I20260929 03:27:33.540859 14654 consensus_queue.cc:237] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } }
I20260929 03:27:33.541501 14135 catalog_manager.cc:5495] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130). New cstate: current_term: 1 leader_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e53e6638536436f9ca6aae65b2bcf4e" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 37211 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 } health_report { overall_health: UNKNOWN } } }
I20260929 03:27:33.547932 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.545308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.6.116.130:37211)
0929 03:27:33.545446 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.545447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.545453 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:27:33.545478 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.545478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.545492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.545578 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:33.545674 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.546147 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.546995 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.547280 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.547374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.547524 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.547679 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.547761 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.547819 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.547834 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260929 03:27:33.552436 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.550196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:56545 (local address 127.6.116.131:45585)
0929 03:27:33.550232 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.550234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.550291 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:27:33.550359 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.550359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.550367 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.550418 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:33.550547 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.551041 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.551804 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.551977 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.552049 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.552212 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.552244 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.552297 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.552362 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.552377 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:33.552492 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.550042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45585 (local address 127.6.116.130:56545)
0929 03:27:33.550240 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:33.550250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:33.550305 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:33.550421 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:33.550421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:33.550422 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:33.550469 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.550470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.551131 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.551132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.551713 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.551715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.552049 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.552050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.552106 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.552119 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.552311 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.552312 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 03:27:33.552358 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:27:33.552393 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
I20260929 03:27:33.554240 14574 raft_consensus.cc:1261] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Refusing update from remote peer 0e53e6638536436f9ca6aae65b2bcf4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:33.554291 14294 raft_consensus.cc:1261] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Refusing update from remote peer 0e53e6638536436f9ca6aae65b2bcf4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:33.554440 14740 consensus_queue.cc:1035] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "afd2385ad7d44521955952b003c0fe0b" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:33.554628 14654 consensus_queue.cc:1035] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e18e717bdd469a9c2b2b7babb0e549" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 41327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:33.561514 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.559308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:54545 (local address 127.6.116.130:37211)
0929 03:27:33.559358 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:33.559360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:33.559403 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:27:33.559462 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:33.559463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:33.559473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:33.559527 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:33.559619 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.560109 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.560822 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.561028 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.561124 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.561277 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.561316 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.561378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:33.561429 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:27:33.561448 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:33.561544 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:33.559258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:37211 (local address 127.6.116.131:54545)
0929 03:27:33.559347 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:33.559356 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:33.559410 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:33.559527 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:33.559528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:33.559529 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 03:27:33.559570 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.559571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.560185 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.560186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.560727 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:33.560729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:33.561133 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:33.561134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:33.561188 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:33.561191 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 03:27:33.561380 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 03:27:33.561382 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:33.561427 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:27:33.561468 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":55,"threads_started":1}
I20260929 03:27:33.568282 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:33.568394 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54176:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:33.568957 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d493630b5504a9a8379931cec67efcc: Sending DeleteTablet for 3 replicas of tablet 7f97c66ea90b4c279675588114374730
I20260929 03:27:33.569205 14274 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f97c66ea90b4c279675588114374730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:59760
I20260929 03:27:33.569240 14553 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f97c66ea90b4c279675588114374730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53240
I20260929 03:27:33.569324 14715 tablet_replica.cc:330] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: stopping tablet replica
I20260929 03:27:33.569332 14414 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f97c66ea90b4c279675588114374730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:33 UTC) from {username='oryx', principal='oryx/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44272
I20260929 03:27:33.569381 14715 raft_consensus.cc:2229] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:33.569412 14715 raft_consensus.cc:2258] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.569427 14712 tablet_replica.cc:330] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: stopping tablet replica
I20260929 03:27:33.569481 14713 tablet_replica.cc:330] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: stopping tablet replica
I20260929 03:27:33.569499 14712 raft_consensus.cc:2229] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:33.569526 14713 raft_consensus.cc:2229] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:33.569556 14713 raft_consensus.cc:2258] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.569574 14712 raft_consensus.cc:2258] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:33.569839 14712 ts_tablet_manager.cc:1905] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.569939 14713 ts_tablet_manager.cc:1905] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.570966 14712 ts_tablet_manager.cc:1918] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:33.571004 14712 log.cc:1196] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/7f97c66ea90b4c279675588114374730
I20260929 03:27:33.571250 14712 ts_tablet_manager.cc:1939] T 7f97c66ea90b4c279675588114374730 P 0e53e6638536436f9ca6aae65b2bcf4e: Deleting consensus metadata
I20260929 03:27:33.571410 14713 ts_tablet_manager.cc:1918] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:33.571458 14713 log.cc:1196] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/7f97c66ea90b4c279675588114374730
I20260929 03:27:33.571483 14121 catalog_manager.cc:4841] TS 0e53e6638536436f9ca6aae65b2bcf4e (127.6.116.130:37211): tablet 7f97c66ea90b4c279675588114374730 (table test-table [id=1c99e0dd1a6542d6bd685c068c56b853]) successfully deleted
I20260929 03:27:33.571656 14713 ts_tablet_manager.cc:1939] T 7f97c66ea90b4c279675588114374730 P afd2385ad7d44521955952b003c0fe0b: Deleting consensus metadata
I20260929 03:27:33.572069 14715 ts_tablet_manager.cc:1905] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:33.573547 14715 ts_tablet_manager.cc:1918] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 03:27:33.573608 14715 log.cc:1196] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/7f97c66ea90b4c279675588114374730
I20260929 03:27:33.573808 14715 ts_tablet_manager.cc:1939] T 7f97c66ea90b4c279675588114374730 P e5e18e717bdd469a9c2b2b7babb0e549: Deleting consensus metadata
I20260929 03:27:33.574208 14122 catalog_manager.cc:4841] TS afd2385ad7d44521955952b003c0fe0b (127.6.116.131:45585): tablet 7f97c66ea90b4c279675588114374730 (table test-table [id=1c99e0dd1a6542d6bd685c068c56b853]) successfully deleted
I20260929 03:27:33.574255 14124 catalog_manager.cc:4841] TS e5e18e717bdd469a9c2b2b7babb0e549 (127.6.116.129:41327): tablet 7f97c66ea90b4c279675588114374730 (table test-table [id=1c99e0dd1a6542d6bd685c068c56b853]) successfully deleted
I20260929 03:27:33.574509 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14209
I20260929 03:27:33.583863 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14349
I20260929 03:27:33.588723 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14489
I20260929 03:27:33.594177 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14105
2026-09-29T03:27:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3296 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:33 dist-test-slave-z2zg krb5kdc[14755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:33 dist-test-slave-z2zg krb5kdc[14755](info): set up 2 sockets
Sep 29 03:27:33 dist-test-slave-z2zg krb5kdc[14755](info): commencing operation
krb5kdc: starting...
W20260929 03:27:34.621294 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:34 dist-test-slave-z2zg krb5kdc[14755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:34.733625 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:37023
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:34.811549 14768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:34.811779 14768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:34.811816 14768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:34.812939 14768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:34.813004 14768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:34.813027 14768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:34.813055 14768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:34.813081 14768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:34.814538 14768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:37023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:34.814800 14768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:34.815003 14768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:34.819392 14776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:34.819520 14768 server_base.cc:1034] running on GCE node
W20260929 03:27:34.819379 14774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:34.819365 14773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:34.819828 14768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:34.820053 14768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:34.821183 14768 hybrid_clock.cc:648] HybridClock initialized: now 1790652454821171 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T03:27:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1235 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:34 dist-test-slave-z2zg krb5kdc[14781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:34 dist-test-slave-z2zg krb5kdc[14781](info): set up 2 sockets
Sep 29 03:27:34 dist-test-slave-z2zg krb5kdc[14781](info): commencing operation
krb5kdc: starting...
W20260929 03:27:35.865268 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:35 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:35.974222 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38281
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:36.051069 14794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.051297 14794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.051332 14794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.052465 14794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:36.052541 14794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.052563 14794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.052584 14794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:36.052614 14794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:36.053978 14794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.054239 14794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.054446 14794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.059096 14800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.059154 14799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.059152 14802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.059286 14794 server_base.cc:1034] running on GCE node
I20260929 03:27:36.059500 14794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.059826 14794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.060986 14794 hybrid_clock.cc:648] HybridClock initialized: now 1790652456060976 us; error 45 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.067557 14794 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:36.067996 14794 webserver.cc:460] Webserver started at http://127.6.116.190:40349/ using document root <none> and password file <none>
I20260929 03:27:36.068219 14794 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.068284 14794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.068444 14794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:36.069478 14794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "347ae940995b4f74818ecf5bfa11a8f9"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "2d286d666ca2241fae5182e9757308ee"
server_key_iv: "04494636e9888d3835ccc984e15728e2"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.069835 14794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "347ae940995b4f74818ecf5bfa11a8f9"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "2d286d666ca2241fae5182e9757308ee"
server_key_iv: "04494636e9888d3835ccc984e15728e2"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.071297 14794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:27:36.071979 14809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.072151 14794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.072243 14794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "347ae940995b4f74818ecf5bfa11a8f9"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "2d286d666ca2241fae5182e9757308ee"
server_key_iv: "04494636e9888d3835ccc984e15728e2"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.072321 14794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.088143 14794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.089727 14794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.089890 14794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.093813 14794 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38281
I20260929 03:27:36.093851 14861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38281 every 8 connection(s)
I20260929 03:27:36.094190 14794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:36.094974 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:36.097195 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Bootstrap starting.
I20260929 03:27:36.097788 14862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:36.097955 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14794
I20260929 03:27:36.098094 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:36.098208 14862 log.cc:824] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:36.098274 6610 external_mini_cluster.cc:1428] Setting key 0702474c46880e35847ba8c35f5922c4
I20260929 03:27:36.098991 14862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: No bootstrap required, opened a new log
I20260929 03:27:36.101629 14862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.101799 14862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:36.101837 14862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347ae940995b4f74818ecf5bfa11a8f9, State: Initialized, Role: FOLLOWER
I20260929 03:27:36.101985 14862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.102070 14862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:36.102123 14862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:36.102169 14862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:36.102739 14862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.102866 14862 leader_election.cc:302] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347ae940995b4f74818ecf5bfa11a8f9; no voters:
I20260929 03:27:36.103080 14862 leader_election.cc:288] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:36.103142 14867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:36.103340 14862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:36.103346 14867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 LEADER]: Becoming Leader. State: Replica: 347ae940995b4f74818ecf5bfa11a8f9, State: Running, Role: LEADER
I20260929 03:27:36.103531 14867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.103868 14869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ae940995b4f74818ecf5bfa11a8f9. Latest consensus state: current_term: 1 leader_uuid: "347ae940995b4f74818ecf5bfa11a8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } } }
I20260929 03:27:36.103999 14869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:36.104396 14876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:36.104876 14876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:36.106099 14868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "347ae940995b4f74818ecf5bfa11a8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } } }
I20260929 03:27:36.106213 14868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:36.106693 14876 catalog_manager.cc:1347] Generated new cluster ID: dcfd7cf7e7af4d81a540c8431d7ad4c2
I20260929 03:27:36.106735 14876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:36.109622 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.098733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.6.116.190:38281)
0929 03:27:36.098894 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.098900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.098923 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:27:36.099473 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.099481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.099537 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.099644 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.100446 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.101287 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.102372 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.102743 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.102814 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.105835 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:36.105839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:36.105840 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:36.105852 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:36.108142 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.108536 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.108539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.108539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.108628 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.108843 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.108844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.108844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.109077 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:36.109222 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:36.109336 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:27:36.109463 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260929 03:27:36.113328 14876 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:36.113965 14876 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:36.117022 14876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Generated new TSK 0
I20260929 03:27:36.117177 14876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:36.142277 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:36.218369 14888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.218544 14888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.218564 14888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.219652 14888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.219686 14888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.219709 14888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:36.221036 14888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.221242 14888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.221448 14888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.227231 14896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.227207 14894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.227203 14893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.227413 14888 server_base.cc:1034] running on GCE node
I20260929 03:27:36.227764 14888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.227983 14888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.229135 14888 hybrid_clock.cc:648] HybridClock initialized: now 1790652456229115 us; error 38 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.231626 14888 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:36.232124 14888 webserver.cc:460] Webserver started at http://127.6.116.129:39059/ using document root <none> and password file <none>
I20260929 03:27:36.232353 14888 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.232412 14888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.232537 14888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:36.233806 14888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "1cee9d4a9f5d41b3b9795d2d59af1751"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "76b0397e0444b99191cc7254f76d1c4b"
server_key_iv: "e0d0f78f1b9a4746ce07a68c52bad260"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.234187 14888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "1cee9d4a9f5d41b3b9795d2d59af1751"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "76b0397e0444b99191cc7254f76d1c4b"
server_key_iv: "e0d0f78f1b9a4746ce07a68c52bad260"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.235491 14888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:36.236162 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.236343 14888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:36.236446 14888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "1cee9d4a9f5d41b3b9795d2d59af1751"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "76b0397e0444b99191cc7254f76d1c4b"
server_key_iv: "e0d0f78f1b9a4746ce07a68c52bad260"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.236536 14888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.254127 14888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.256000 14888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.256160 14888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.256374 14888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:36.256670 14888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:36.256726 14888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.256770 14888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:36.256814 14888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.262900 14888 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:43563
I20260929 03:27:36.262948 15016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:43563 every 8 connection(s)
I20260929 03:27:36.263268 14888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:36.266155 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 14888
I20260929 03:27:36.266279 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:36.266474 6610 external_mini_cluster.cc:1428] Setting key 5c9a13542e6e93bbbbe6587edd473661
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:36.272727 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.263980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:46997 (local address 127.6.116.190:38281)
0929 03:27:36.264027 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.264029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.264247 (+ 218us) server_negotiation.cc:407] Connection header received
0929 03:27:36.264665 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.264666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.264676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.264747 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.265590 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.266082 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.267545 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.267727 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.267779 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.270058 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:36.270059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:36.270060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:36.270067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:36.271471 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.271865 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.271867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.271867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.271922 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.272167 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.272168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.272168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.272267 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:36.272334 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:36.272573 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:27:36.272644 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:27:36.272891 15019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.263970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:46997)
0929 03:27:36.264140 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.264170 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.264581 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.264816 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.264819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.265328 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.265486 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.265490 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.266187 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.266188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.267472 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.267475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.267803 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.267804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.267938 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.268490 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.268500 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.269968 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:36.271554 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.271561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.271563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.271777 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.271985 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.271986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.271986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.272090 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.272328 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:36.272334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:36.272454 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 03:27:36.272544 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:27:36.272686 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260929 03:27:36.273337 15017 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38281
I20260929 03:27:36.273407 15017 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:36.273562 15017 heartbeater.cc:511] Master 127.6.116.190:38281 requested a full tablet report, sending...
I20260929 03:27:36.274046 14826 ts_manager.cc:194] Registered new tserver with Master: 1cee9d4a9f5d41b3b9795d2d59af1751 (127.6.116.129:43563)
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:36.274655 14826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:46997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:36.289449 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:36.368981 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.369184 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.369220 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.370334 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.370400 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.370446 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:36.371869 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.372104 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.372320 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.377149 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.377151 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.377269 15022 server_base.cc:1034] running on GCE node
W20260929 03:27:36.377178 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.377568 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.377811 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.378968 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790652456378955 us; error 24 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.381093 15022 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:36.381460 15022 webserver.cc:460] Webserver started at http://127.6.116.130:34889/ using document root <none> and password file <none>
I20260929 03:27:36.381681 15022 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.381747 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.381886 15022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:36.382807 15022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "ba3afb797f42456d885489aa4660d202"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "5fa4d5806c5141831e3f320a02cd8d26"
server_key_iv: "64166b0652af241c4f9c0f0646a6f1e0"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.383206 15022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "ba3afb797f42456d885489aa4660d202"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "5fa4d5806c5141831e3f320a02cd8d26"
server_key_iv: "64166b0652af241c4f9c0f0646a6f1e0"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.384397 15022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:36.385051 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.385221 15022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:36.385304 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "ba3afb797f42456d885489aa4660d202"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "5fa4d5806c5141831e3f320a02cd8d26"
server_key_iv: "64166b0652af241c4f9c0f0646a6f1e0"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.385385 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.394137 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.395865 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.396011 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.396215 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:36.396479 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:36.396531 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.396567 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:36.396605 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.402040 15022 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:37627
I20260929 03:27:36.402103 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:37627 every 8 connection(s)
I20260929 03:27:36.402371 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:36.403522 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15022
I20260929 03:27:36.403599 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:36.403786 6610 external_mini_cluster.cc:1428] Setting key 758effaa467b6ba93415182028e7a70c
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:36.412067 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.402981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:35649 (local address 127.6.116.190:38281)
0929 03:27:36.403021 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.403024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.403234 (+ 210us) server_negotiation.cc:407] Connection header received
0929 03:27:36.403725 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.403726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.403740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.403800 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.404852 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.405392 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.406890 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.407134 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.407187 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.409397 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:36.409398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:36.409398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:36.409404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:36.410676 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.411136 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.411137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.411137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.411185 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.411460 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.411460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.411460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.411545 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:36.411589 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:36.411935 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 03:27:36.412009 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:36.412277 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.402946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:35649)
0929 03:27:36.403121 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.403160 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.403602 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.403858 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.403862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.404598 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.404764 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.404768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.405475 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.405476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.406793 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.406796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.407210 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.407211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.407349 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.407751 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.407760 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.409309 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:36.410754 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.410761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.410762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.411036 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.411250 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.411251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.411252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.411397 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.411632 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:36.411637 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:36.411792 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 03:27:36.411919 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:27:36.412080 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:27:36.412738 15151 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38281
I20260929 03:27:36.412832 15151 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:36.412984 15151 heartbeater.cc:511] Master 127.6.116.190:38281 requested a full tablet report, sending...
I20260929 03:27:36.413365 14826 ts_manager.cc:194] Registered new tserver with Master: ba3afb797f42456d885489aa4660d202 (127.6.116.130:37627)
I20260929 03:27:36.413717 14826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:35649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:36.429203 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:36.504123 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.504369 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.504405 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.505571 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.505640 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.505702 15156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:36.507125 15156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.507345 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.507571 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.511909 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.511888 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.511911 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.512123 15156 server_base.cc:1034] running on GCE node
I20260929 03:27:36.512359 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.512585 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.513741 15156 hybrid_clock.cc:648] HybridClock initialized: now 1790652456513730 us; error 27 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.516080 15156 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:36.516508 15156 webserver.cc:460] Webserver started at http://127.6.116.131:44991/ using document root <none> and password file <none>
I20260929 03:27:36.516716 15156 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.516780 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.516937 15156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:36.517906 15156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "04eb700958274a1c9fd2f92af91cd717"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "f36488e481bc3afdfc332e1289c5875a"
server_key_iv: "072b1358f1431486c1537e8eec47906c"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.518251 15156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "04eb700958274a1c9fd2f92af91cd717"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "f36488e481bc3afdfc332e1289c5875a"
server_key_iv: "072b1358f1431486c1537e8eec47906c"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.519519 15156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:36.520148 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.520337 15156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:36.520426 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "04eb700958274a1c9fd2f92af91cd717"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "f36488e481bc3afdfc332e1289c5875a"
server_key_iv: "072b1358f1431486c1537e8eec47906c"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.520499 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.541935 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.544160 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.544317 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.544518 15156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:36.544791 15156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:36.544843 15156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.544875 15156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:36.544903 15156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.550423 15156 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:46723
I20260929 03:27:36.550465 15284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:46723 every 8 connection(s)
I20260929 03:27:36.550726 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:36.553201 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15156
I20260929 03:27:36.553309 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:36.553486 6610 external_mini_cluster.cc:1428] Setting key d94ea2ceab9610d7d6190438a3efad70
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:36.559000 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.551300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:56059 (local address 127.6.116.190:38281)
0929 03:27:36.551377 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.551379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.551517 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:27:36.551946 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.551947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.551959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.552000 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.552873 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.553550 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.554613 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.554820 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.554907 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.556791 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:36.556792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:36.556792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:36.556798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:36.557961 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.558293 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.558294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.558294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.558339 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:36.558532 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:36.558533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:36.558533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:36.558634 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:36.558682 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:36.558885 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 03:27:36.558940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 03:27:36.559156 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.551254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:56059)
0929 03:27:36.551414 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.551446 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.551844 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.552070 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.552073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.552614 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.552777 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.552781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.553644 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.553645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.554525 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.554527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.554917 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.554918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.555028 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.555407 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.555414 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.556710 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:36.558027 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.558032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.558033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.558231 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.558394 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:36.558394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:36.558395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:36.558481 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:36.558686 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:36.558689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:36.558784 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 03:27:36.558860 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 03:27:36.558961 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260929 03:27:36.559501 15285 heartbeater.cc:347] Connected to a master server at 127.6.116.190:38281
I20260929 03:27:36.559614 15285 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:36.559789 15285 heartbeater.cc:511] Master 127.6.116.190:38281 requested a full tablet report, sending...
I20260929 03:27:36.560127 14826 ts_manager.cc:194] Registered new tserver with Master: 04eb700958274a1c9fd2f92af91cd717 (127.6.116.131:46723)
I20260929 03:27:36.560487 14826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:56059
I20260929 03:27:36.565030 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
Entry for principal oryx/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab.
WARNING: no policy specified for oryx/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
Entry for principal oryx/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab.
WARNING: no policy specified for oryx/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
Entry for principal oryx/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab.
I20260929 03:27:36.629685 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14888
I20260929 03:27:36.634050 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15022
I20260929 03:27:36.638181 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15156
I20260929 03:27:36.642184 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 14794
I20260929 03:27:36.647203 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38281
--webserver_interface=127.6.116.190
--webserver_port=40349
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:36.724224 15294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.724393 15294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.724414 15294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.725420 15294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:36.725450 15294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.725463 15294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.725476 15294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:36.725487 15294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:36.726822 15294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=40349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.726984 15294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.727191 15294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.731493 15299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.731600 15294 server_base.cc:1034] running on GCE node
W20260929 03:27:36.731563 15300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.731679 15302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.731930 15294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.732198 15294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.733346 15294 hybrid_clock.cc:648] HybridClock initialized: now 1790652456733338 us; error 36 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.735669 15294 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:36.736109 15294 webserver.cc:460] Webserver started at http://127.6.116.190:40349/ using document root <none> and password file <none>
I20260929 03:27:36.736315 15294 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.736398 15294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.737609 15294 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:36.738179 15309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.738332 15294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.738423 15294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "347ae940995b4f74818ecf5bfa11a8f9"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "2d286d666ca2241fae5182e9757308ee"
server_key_iv: "04494636e9888d3835ccc984e15728e2"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.738723 15294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.752913 15294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.755038 15294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.755213 15294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.758853 15294 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38281
I20260929 03:27:36.758888 15361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38281 every 8 connection(s)
I20260929 03:27:36.759222 15294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:36.760479 15362 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:27:36.761171 15362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Bootstrap starting.
I20260929 03:27:36.761186 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15294
I20260929 03:27:36.761471 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:43563
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:38281
--webserver_port=39059
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:27:36.769507 15362 log.cc:824] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:36.770969 15362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:27:36.771266 15362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Bootstrap complete.
I20260929 03:27:36.773355 15362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.773551 15362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 347ae940995b4f74818ecf5bfa11a8f9, State: Initialized, Role: FOLLOWER
I20260929 03:27:36.773746 15362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.773800 15362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:36.773859 15362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:36.773904 15362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:27:36.774914 15362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.774993 15362 leader_election.cc:302] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 347ae940995b4f74818ecf5bfa11a8f9; no voters:
I20260929 03:27:36.775202 15362 leader_election.cc:288] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:27:36.775231 15366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:27:36.775337 15366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [term 2 LEADER]: Becoming Leader. State: Replica: 347ae940995b4f74818ecf5bfa11a8f9, State: Running, Role: LEADER
I20260929 03:27:36.775425 15366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } }
I20260929 03:27:36.775449 15362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:36.775683 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 347ae940995b4f74818ecf5bfa11a8f9. Latest consensus state: current_term: 2 leader_uuid: "347ae940995b4f74818ecf5bfa11a8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } } }
I20260929 03:27:36.775863 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:36.775693 15367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "347ae940995b4f74818ecf5bfa11a8f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "347ae940995b4f74818ecf5bfa11a8f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38281 } } }
I20260929 03:27:36.776088 15367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:36.776263 15370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:36.776760 15370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:36.776959 15370 catalog_manager.cc:1259] Loaded cluster ID: dcfd7cf7e7af4d81a540c8431d7ad4c2
I20260929 03:27:36.777141 15370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:36.777915 15370 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:36.778041 15370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 347ae940995b4f74818ecf5bfa11a8f9: Loaded TSK: 0
I20260929 03:27:36.778208 15370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:27:36.853480 15364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.853667 15364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.853688 15364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.854871 15364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.854910 15364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.854935 15364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:36.856462 15364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.129.keytab
--principal=oryx/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:43563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=39059
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.856675 15364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.856914 15364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.862500 15389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.862560 15388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.862679 15364 server_base.cc:1034] running on GCE node
W20260929 03:27:36.862507 15391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.862960 15364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.863262 15364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.864423 15364 hybrid_clock.cc:648] HybridClock initialized: now 1790652456864407 us; error 30 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.867239 15364 init.cc:377] Logged in from keytab as oryx/127.6.116.129@KRBTEST.COM (short username oryx)
I20260929 03:27:36.867717 15364 webserver.cc:460] Webserver started at http://127.6.116.129:39059/ using document root <none> and password file <none>
I20260929 03:27:36.867946 15364 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.868032 15364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.869673 15364 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:36.870349 15398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.870601 15364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:36.870694 15364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "1cee9d4a9f5d41b3b9795d2d59af1751"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "76b0397e0444b99191cc7254f76d1c4b"
server_key_iv: "e0d0f78f1b9a4746ce07a68c52bad260"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.871081 15364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:36.883081 15364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:36.885190 15364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:36.885324 15364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:36.885525 15364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:36.885833 15364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:36.885874 15364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.885901 15364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:36.885929 15364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.893143 15364 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:43563
I20260929 03:27:36.893225 15511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:43563 every 8 connection(s)
I20260929 03:27:36.893543 15364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:36.896348 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15364
I20260929 03:27:36.896632 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:37627
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:38281
--webserver_port=34889
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:36.903209 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.894248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:38817)
0929 03:27:36.894419 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.894460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.894868 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.895328 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.895331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.895848 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.896014 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.896018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.897676 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.897676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.898882 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.898884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.899776 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.899777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.899903 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.900449 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.900458 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.902986 (+ 2528us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
W20260929 03:27:36.903412 15512 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:38281 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.116.190:38281: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
W20260929 03:27:36.903676 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:36.894319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:38817 (local address 127.6.116.190:38281)
0929 03:27:36.894515 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.894523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.894561 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:27:36.895096 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.895103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.895160 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.895294 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.896142 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.897557 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.898977 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.899598 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.899694 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.903485 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:38817: BlockingRecv error: recv got EOF from 127.6.116.129:38817 (error 108)
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:27:36.908830 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:36.903684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:40637 (local address 127.6.116.190:38281)
0929 03:27:36.903764 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.903766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.903915 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:27:36.903934 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.903935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.903945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.903987 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.904333 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.905026 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.905979 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.906293 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.906359 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.908777 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:40637: BlockingRecv error: recv got EOF from 127.6.116.129:40637 (error 108)
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":59,"threads_started":1}
I20260929 03:27:36.908980 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.903641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:40637)
0929 03:27:36.903812 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.903823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.903868 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.904032 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.904033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.904203 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.904250 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.904251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.905122 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.905123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.905884 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.905887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.906410 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.906411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.906474 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.906881 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.906886 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.908920 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:36.914084 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:36.909321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:56045)
0929 03:27:36.909409 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:36.909419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:36.909461 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:36.909590 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:36.909591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:36.909744 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:36.909773 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.909774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.910630 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.910630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.911409 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:36.911411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.911864 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.911865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:36.911925 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.912327 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:36.912332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:36.914025 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:27:36.914165 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:36.909381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:56045 (local address 127.6.116.190:38281)
0929 03:27:36.909416 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:36.909418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:36.909451 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:27:36.909508 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:36.909509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:36.909518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:36.909580 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:36.909848 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.910538 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.911492 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:36.911785 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:36.911854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:36.914114 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:56045: BlockingRecv error: recv got EOF from 127.6.116.129:56045 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 03:27:36.914343 15512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:27:36.976707 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:36.976930 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:36.976965 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:36.978060 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:36.978120 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:36.978165 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:36.979542 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.130.keytab
--principal=oryx/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:37627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=34889
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:36.979780 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:36.979986 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:36.984573 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.984597 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:36.984587 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:36.984962 15516 server_base.cc:1034] running on GCE node
I20260929 03:27:36.985095 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:36.985249 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:36.986397 15516 hybrid_clock.cc:648] HybridClock initialized: now 1790652456986386 us; error 24 us; skew 500 ppm
Sep 29 03:27:36 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652456, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:36.988792 15516 init.cc:377] Logged in from keytab as oryx/127.6.116.130@KRBTEST.COM (short username oryx)
I20260929 03:27:36.989130 15516 webserver.cc:460] Webserver started at http://127.6.116.130:34889/ using document root <none> and password file <none>
I20260929 03:27:36.989295 15516 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:36.989343 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:36.990617 15516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:36.991263 15533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:36.991436 15516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:36.991524 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "ba3afb797f42456d885489aa4660d202"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "5fa4d5806c5141831e3f320a02cd8d26"
server_key_iv: "64166b0652af241c4f9c0f0646a6f1e0"
server_key_version: "encryptionkey@0"
I20260929 03:27:36.991848 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:37.012271 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:37.014326 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:37.014472 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:37.014672 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:37.014950 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:37.015000 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:37.015054 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:37.015086 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:37.020450 15516 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:37627
I20260929 03:27:37.020529 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:37627 every 8 connection(s)
I20260929 03:27:37.020805 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:37.021051 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15516
I20260929 03:27:37.021350 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:46723
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:38281
--webserver_port=44991
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.027954 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.021472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:47937)
0929 03:27:37.021622 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.021660 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.022109 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.022344 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.022348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.022842 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.023001 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.023005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.023902 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.023903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.024729 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.024731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.025221 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.025222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.025312 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.025785 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.025792 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.027837 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260929 03:27:37.028023 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.021500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:47937 (local address 127.6.116.190:38281)
0929 03:27:37.021542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.021543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.021744 (+ 201us) server_negotiation.cc:407] Connection header received
0929 03:27:37.022218 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.022219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.022235 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.022286 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.023140 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.023793 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.024814 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.025140 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.025219 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.027943 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:47937: BlockingRecv error: recv got EOF from 127.6.116.130:47937 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:27:37.028179 15647 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:38281 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.116.190:38281: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.032266 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.028389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:46667)
0929 03:27:37.028433 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.028443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.028482 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.028594 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.028595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.028713 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.028748 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.028749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.029360 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.029361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.029848 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.029849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.030214 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.030215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.030277 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.030690 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.030693 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.032208 (+ 1515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:27:37.032366 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.028445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:46667 (local address 127.6.116.190:38281)
0929 03:27:37.028505 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.028507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.028516 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:37.028538 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.028539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.028547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.028601 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.028847 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.029286 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.029939 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.030134 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.030213 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.032284 (+ 2071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:46667: BlockingRecv error: recv got EOF from 127.6.116.130:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.036286 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.032627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:36019)
0929 03:27:37.032670 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.032679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.032734 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.032839 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.032840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.032951 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.032972 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.032973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.033525 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.033526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.033973 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.033974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.034323 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.034323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.034371 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.034743 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.034748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.036217 (+ 1469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:27:37.036336 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.032688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:36019 (local address 127.6.116.190:38281)
0929 03:27:37.032739 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.032741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.032758 (+ 17us) server_negotiation.cc:407] Connection header received
0929 03:27:37.032777 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.032777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.032786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.032836 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.033047 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.033454 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.034070 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.034244 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.034321 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.036278 (+ 1957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:36019: BlockingRecv error: recv got EOF from 127.6.116.130:36019 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 03:27:37.036521 15647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:27:37.100042 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:37.100226 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:37.100248 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:37.101521 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:37.101562 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:37.101605 15650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:37.102942 15650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/krb5kdc/oryx_127.6.116.131.keytab
--principal=oryx/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:46723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=44991
--tserver_master_addrs=127.6.116.190:38281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:37.103176 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:37.103396 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:37.108026 15655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:37.108006 15656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:37.107995 15658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:37.108270 15650 server_base.cc:1034] running on GCE node
I20260929 03:27:37.108560 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:37.108781 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:37.109946 15650 hybrid_clock.cc:648] HybridClock initialized: now 1790652457109923 us; error 30 us; skew 500 ppm
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652457, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:37.112164 15650 init.cc:377] Logged in from keytab as oryx/127.6.116.131@KRBTEST.COM (short username oryx)
I20260929 03:27:37.112577 15650 webserver.cc:460] Webserver started at http://127.6.116.131:44991/ using document root <none> and password file <none>
I20260929 03:27:37.112799 15650 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:37.112876 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:37.114076 15650 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:37.114569 15665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:37.114714 15650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:37.114802 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "04eb700958274a1c9fd2f92af91cd717"
format_stamp: "Formatted at 2026-09-29 03:27:36 on dist-test-slave-z2zg"
server_key: "f36488e481bc3afdfc332e1289c5875a"
server_key_iv: "072b1358f1431486c1537e8eec47906c"
server_key_version: "encryptionkey@0"
I20260929 03:27:37.115130 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:37.136147 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:37.137751 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:37.137897 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:37.138099 15650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:37.138365 15650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:37.138414 15650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:37.138453 15650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:37.138478 15650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:37.144119 15650 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:46723
I20260929 03:27:37.144164 15778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:46723 every 8 connection(s)
I20260929 03:27:37.144486 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:37.146368 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15650
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:27:37.152132 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.145087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:38885 (local address 127.6.116.190:38281)
0929 03:27:37.145145 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.145147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.145318 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:27:37.145758 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.145759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.145768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.145820 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.146860 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.147472 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.148618 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.148790 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.148836 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.152088 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:38885: BlockingRecv error: recv got EOF from 127.6.116.131:38885 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:37.152187 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.145044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:38885)
0929 03:27:37.145191 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.145226 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.145671 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.145866 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.145869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.146431 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.146623 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.146628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.147562 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.147563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.148532 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.148535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.148867 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.148868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.148994 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.149479 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.149488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.152011 (+ 2523us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
W20260929 03:27:37.152359 15779 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:38281 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.116.190:38281: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
I20260929 03:27:37.153825 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.146879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.6.116.190:38281)
0929 03:27:37.146955 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.146958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.146966 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:37.147020 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.147021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.147031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.147125 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.147700 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.148154 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.149030 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.149197 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.149281 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.150534 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:37.150538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:37.150538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:37.150555 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:37.152454 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:37.152855 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:37.152857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:37.152858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:37.152933 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:37.153149 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:37.153149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:37.153150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:37.153426 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:37.153545 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:37.153632 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 03:27:37.153733 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.156860 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.152588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:33023)
0929 03:27:37.152625 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.152633 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.152666 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.152775 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.152776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.152899 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.152944 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.152945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.153819 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.153820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.154399 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.154401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.154740 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.154741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.154799 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.155192 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.155196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.156821 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:27:37.156996 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.152618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:33023 (local address 127.6.116.190:38281)
0929 03:27:37.152654 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.152656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.152663 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:37.152712 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.152713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.152719 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.152754 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.153014 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.153679 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.154468 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.154664 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.154730 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.156914 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:33023: BlockingRecv error: recv got EOF from 127.6.116.131:33023 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.160912 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.157240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:53099)
0929 03:27:37.157290 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.157300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.157344 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.157467 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.157468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.157571 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.157602 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.157603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.158124 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.158125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.158564 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.158565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.158858 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.158859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.158890 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.159450 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.159453 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.160843 (+ 1390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:27:37.160951 15515 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.157299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:53099 (local address 127.6.116.190:38281)
0929 03:27:37.157363 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.157364 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.157388 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:27:37.157406 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.157406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.157413 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.157457 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.157658 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.158064 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.158626 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.158803 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.158847 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.160899 (+ 2052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:53099: BlockingRecv error: recv got EOF from 127.6.116.131:53099 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 03:27:37.161054 15779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:37 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:37.919062 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:37.914698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:49435)
0929 03:27:37.914815 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:37.914831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:37.914875 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:37.915009 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:37.915010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:37.915176 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:37.915221 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.915223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.915847 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.915847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.916379 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:37.916381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.916799 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.916800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:37.916869 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.917346 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:37.917350 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:37.918979 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
W20260929 03:27:37.919219 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:37.914771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:49435 (local address 127.6.116.190:38281)
0929 03:27:37.914899 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:37.914901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:37.914908 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:37.914928 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:37.914928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:37.914937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:37.914995 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:37.915280 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.915750 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.916445 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:37.916699 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:37.916776 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:37.919141 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:49435: BlockingRecv error: recv got EOF from 127.6.116.129:49435 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
W20260929 03:27:38.041029 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:38.036900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:51335 (local address 127.6.116.190:38281)
0929 03:27:38.036966 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:38.036969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:38.037071 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:27:38.037133 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:38.037133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:38.037143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:38.037187 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:38.037447 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.037927 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.038578 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.038754 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.038801 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.040985 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:51335: BlockingRecv error: recv got EOF from 127.6.116.130:51335 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:27:38.040958 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:38.036854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:51335)
0929 03:27:38.036999 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:38.037010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:38.037065 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:38.037202 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:38.037203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:38.037344 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:38.037388 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.037390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.037986 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.037987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.038512 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.038513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.038819 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.038820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.038856 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.039263 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:38.039267 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:38.040899 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:38.165871 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:38.161567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:35605)
0929 03:27:38.161702 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:38.161712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:38.161768 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:38.161886 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:38.161887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:38.162038 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:38.162081 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.162083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.162795 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.162796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.163328 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.163330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.163678 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.163679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.163742 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.164133 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:38.164137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:38.165773 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260929 03:27:38.165917 15784 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:38.161667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:35605 (local address 127.6.116.190:38281)
0929 03:27:38.161717 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:38.161718 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:38.161772 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:27:38.161821 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:38.161821 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:38.161832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:38.161869 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:38.162164 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.162715 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.163421 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.163609 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.163679 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.165854 (+ 2175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:35605: BlockingRecv error: recv got EOF from 127.6.116.131:35605 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:38 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:38.923820 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:38.919484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:53325)
0929 03:27:38.919590 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:38.919603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:38.919646 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:38.919787 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:38.919787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:38.919925 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:38.919969 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.919971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.920548 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.920549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.921112 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:38.921113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.921409 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.921410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:38.921446 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.921884 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:38.921889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:38.923742 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
W20260929 03:27:38.923910 15788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:38.919551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:53325 (local address 127.6.116.190:38281)
0929 03:27:38.919684 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:38.919685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:38.919692 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:38.919711 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:38.919711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:38.919720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:38.919775 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:38.920025 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.920479 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.921176 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:38.921352 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:38.921397 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:38.923824 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:53325: BlockingRecv error: recv got EOF from 127.6.116.129:53325 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:39.045800 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:39.041470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:58281)
0929 03:27:39.041576 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:39.041587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:39.041649 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:39.041771 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:39.041772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:39.041957 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:39.042001 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.042003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.042718 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.042719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.043226 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.043228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.043535 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.043536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.043571 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.043980 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:39.043985 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:39.045700 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":54,"threads_started":1}
W20260929 03:27:39.045858 15788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:39.041537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:58281 (local address 127.6.116.190:38281)
0929 03:27:39.041608 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:39.041612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:39.041637 (+ 25us) server_negotiation.cc:407] Connection header received
0929 03:27:39.041700 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:39.041700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:39.041710 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:39.041749 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:39.042078 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.042656 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.043297 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.043472 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.043529 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.045773 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:58281: BlockingRecv error: recv got EOF from 127.6.116.130:58281 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:39.170950 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:39.166330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:51571)
0929 03:27:39.166462 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:39.166471 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:39.166541 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:39.166687 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:39.166687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:39.166846 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:39.166889 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.166890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.167624 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.167626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.168140 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.168141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.168505 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.168506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.168566 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.168973 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:39.168977 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:39.170901 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
W20260929 03:27:39.171085 15788 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:39.166460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51571 (local address 127.6.116.190:38281)
0929 03:27:39.166528 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:39.166530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:39.166539 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:39.166613 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:39.166614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:39.166628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:39.166672 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:39.166970 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.167532 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.168229 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.168424 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.168503 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.170975 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:51571: BlockingRecv error: recv got EOF from 127.6.116.131:51571 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:39 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:39.928678 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:39.924299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:34243)
0929 03:27:39.924438 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:39.924447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:39.924492 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:39.924629 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:39.924630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:39.924778 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:39.924825 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.924826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.925435 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.925435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.925943 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:39.925944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.926284 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.926284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:39.926324 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.926853 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:39.926859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:39.928623 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
W20260929 03:27:39.928800 15792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:39.924380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34243 (local address 127.6.116.190:38281)
0929 03:27:39.924494 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:39.924496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:39.924525 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:27:39.924544 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:39.924544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:39.924554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:39.924601 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:39.924888 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.925372 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.926025 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:39.926222 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:39.926270 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:39.928715 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:34243: BlockingRecv error: recv got EOF from 127.6.116.129:34243 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:40.050521 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:40.046249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:57835)
0929 03:27:40.046388 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:40.046398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:40.046450 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:40.046572 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:40.046573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:40.046726 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:40.046789 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.046791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.047414 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.047414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.047932 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.047934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.048242 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.048243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.048290 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.048782 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:40.048787 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:40.050439 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260929 03:27:40.050628 15792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:40.046324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:57835 (local address 127.6.116.190:38281)
0929 03:27:40.046388 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:40.046390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:40.046438 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:27:40.046504 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:40.046505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:40.046515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:40.046554 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:40.046851 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.047353 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.047996 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.048182 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.048229 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.050544 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:57835: BlockingRecv error: recv got EOF from 127.6.116.130:57835 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:40.175730 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:40.171348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:49309)
0929 03:27:40.171479 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:40.171491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:40.171559 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:40.171749 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:40.171750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:40.171963 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:40.172017 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.172021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.172623 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.172624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.173119 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.173120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.173423 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.173424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.173460 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.173913 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:40.173920 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:40.175654 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":120,"threads_started":1}
W20260929 03:27:40.175756 15792 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:40.171435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:49309 (local address 127.6.116.190:38281)
0929 03:27:40.171482 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:40.171485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:40.171548 (+ 63us) server_negotiation.cc:407] Connection header received
0929 03:27:40.171659 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:40.171660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:40.171674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:40.171728 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:40.172101 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.172559 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.173191 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.173348 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.173411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.175713 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:49309: BlockingRecv error: recv got EOF from 127.6.116.131:49309 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:40 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:40.933727 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:40.929261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:45167)
0929 03:27:40.929401 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:40.929428 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:40.929496 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:40.929729 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:40.929730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:40.929942 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:40.930001 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.930003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.930649 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.930650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.931280 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:40.931282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.931603 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.931604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:40.931642 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.932050 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:40.932055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:40.933671 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260929 03:27:40.933840 15796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:40.929318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:45167 (local address 127.6.116.190:38281)
0929 03:27:40.929444 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:40.929446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:40.929576 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:27:40.929621 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:40.929622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:40.929634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:40.929704 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:40.930089 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.930580 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.931358 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:40.931540 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:40.931602 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:40.933756 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:45167: BlockingRecv error: recv got EOF from 127.6.116.129:45167 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":88,"threads_started":1}
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:41.055356 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:41.051117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:46717)
0929 03:27:41.051244 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:41.051257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:41.051318 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:41.051443 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:41.051444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:41.051596 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:41.051642 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.051644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.052257 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.052258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.052771 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.052772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.053083 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.053084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.053123 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.053533 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:41.053537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:41.055253 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
W20260929 03:27:41.055394 15796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:41.051159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:46717 (local address 127.6.116.190:38281)
0929 03:27:41.051241 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:41.051243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:41.051323 (+ 80us) server_negotiation.cc:407] Connection header received
0929 03:27:41.051364 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:41.051365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:41.051375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:41.051427 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:41.051705 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.052191 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.052838 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.053022 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.053071 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.055338 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:46717: BlockingRecv error: recv got EOF from 127.6.116.130:46717 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:41.183108 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:41.176149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:39305)
0929 03:27:41.176309 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:41.176325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:41.176402 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:41.176603 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:41.176604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:41.176801 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:41.176852 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.176853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.177583 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.177584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.178115 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.178117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.178474 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.178475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.178534 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.181442 (+ 2908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:41.181446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:41.182982 (+ 1536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
W20260929 03:27:41.183144 15796 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:41.176227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39305 (local address 127.6.116.190:38281)
0929 03:27:41.176282 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:41.176284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:41.176383 (+ 99us) server_negotiation.cc:407] Connection header received
0929 03:27:41.176515 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:41.176516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:41.176526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:41.176580 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:41.176932 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.177510 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.178199 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.178392 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.178480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.183086 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:39305: BlockingRecv error: recv got EOF from 127.6.116.131:39305 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:41 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:41.938297 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:41.934178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:36801)
0929 03:27:41.934307 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:41.934318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:41.934370 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:41.934510 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:41.934511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:41.934661 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:41.934720 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.934722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.935319 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.935320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.935794 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:41.935796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.936098 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.936099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:41.936155 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.936572 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:41.936578 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:41.938242 (+ 1664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
W20260929 03:27:41.938407 15800 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:41.934255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:36801 (local address 127.6.116.190:38281)
0929 03:27:41.934364 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:41.934366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:41.934374 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:41.934418 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:41.934418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:41.934428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:41.934486 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:41.934780 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.935254 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.935862 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:41.936034 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:41.936087 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:41.938331 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:36801: BlockingRecv error: recv got EOF from 127.6.116.129:36801 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:42.060087 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:42.055803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:55985)
0929 03:27:42.055941 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:42.055951 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:42.056001 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:42.056128 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:42.056128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:42.056280 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:42.056326 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.056327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.056941 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.056941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.057461 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.057462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.057776 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.057777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.057813 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.058259 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:42.058263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:42.060002 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":53,"threads_started":1}
W20260929 03:27:42.060197 15800 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:42.055911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55985 (local address 127.6.116.190:38281)
0929 03:27:42.055977 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:42.055979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:42.055998 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:27:42.056049 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:42.056050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:42.056060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:42.056107 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:42.056385 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.056879 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.057539 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.057716 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.057765 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.060139 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:55985: BlockingRecv error: recv got EOF from 127.6.116.130:55985 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:42.188258 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:42.183642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:59695)
0929 03:27:42.183796 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:42.183807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:42.183894 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:42.184060 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:42.184061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:42.184233 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:42.184277 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.184278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.184901 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.184902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.185423 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.185425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.185778 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.185779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.185842 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.186295 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:42.186299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:42.188175 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260929 03:27:42.188313 15800 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:42.183713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:59695 (local address 127.6.116.190:38281)
0929 03:27:42.183792 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:42.183795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:42.183878 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:27:42.183966 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:42.183967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:42.183979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:42.184052 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:42.184352 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.184825 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.185512 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.185700 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.185777 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.188267 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:59695: BlockingRecv error: recv got EOF from 127.6.116.131:59695 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:42 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:42.943396 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:42.938846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:42287)
0929 03:27:42.938974 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:42.938984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:42.939029 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:42.939159 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:42.939160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:42.939333 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:42.939378 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.939380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.939972 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.939973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.940504 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:42.940506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.940886 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.940887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:42.940937 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.941406 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:42.941411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:42.943301 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
W20260929 03:27:42.943445 15804 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:42.938883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:42287 (local address 127.6.116.190:38281)
0929 03:27:42.939012 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:42.939014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:42.939021 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:42.939091 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:42.939092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:42.939101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:42.939155 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:42.939434 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.939907 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.940572 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:42.940797 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:42.940865 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:42.943376 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:42287: BlockingRecv error: recv got EOF from 127.6.116.129:42287 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:43.065217 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:43.060636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:51225)
0929 03:27:43.060812 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:43.060824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:43.060915 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:43.061066 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:43.061068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:43.061261 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:43.061322 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.061324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.061995 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.061996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.062518 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.062520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.062889 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.062890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.062966 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.063424 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:43.063428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:43.065099 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
W20260929 03:27:43.065238 15804 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:43.060722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:51225 (local address 127.6.116.190:38281)
0929 03:27:43.060797 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:43.060800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:43.060897 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:27:43.060972 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:43.060973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:43.060986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:43.061058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:43.061403 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.061903 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.062614 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.062807 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.062877 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.065179 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:51225: BlockingRecv error: recv got EOF from 127.6.116.130:51225 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:43.193238 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:43.188729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:49701)
0929 03:27:43.188911 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:43.188921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:43.188989 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:43.189107 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:43.189108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:43.189283 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:43.189325 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.189327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.189995 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.189996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.190521 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.190522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.190904 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.190905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.190969 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.191410 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:43.191415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:43.193142 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260929 03:27:43.193282 15804 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:43.188826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:49701 (local address 127.6.116.190:38281)
0929 03:27:43.188909 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:43.188911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:43.188972 (+ 61us) server_negotiation.cc:407] Connection header received
0929 03:27:43.189041 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:43.189042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:43.189056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:43.189104 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:43.189402 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.189914 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.190620 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.190813 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.190902 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.193220 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:49701: BlockingRecv error: recv got EOF from 127.6.116.131:49701 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:43 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:43.948145 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:43.943837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:60981)
0929 03:27:43.943982 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:43.943996 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:43.944061 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:43.944213 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:43.944214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:43.944371 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:43.944419 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.944420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.945051 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.945052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.945563 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:43.945564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.945876 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.945877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:43.945913 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.946350 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:43.946354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:43.948032 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
W20260929 03:27:43.948240 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:43.943891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:60981 (local address 127.6.116.190:38281)
0929 03:27:43.944038 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:43.944040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:43.944048 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:43.944126 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:43.944127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:43.944137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:43.944193 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:43.944490 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.944989 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.945631 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:43.945814 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:43.945863 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:43.948157 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:60981: BlockingRecv error: recv got EOF from 127.6.116.129:60981 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:44.070281 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:44.065761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:41515)
0929 03:27:44.065906 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:44.065917 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:44.065998 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:44.066196 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:44.066197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:44.066401 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:44.066468 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.066470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.067161 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.067162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.067715 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.067716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.068033 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.068034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.068073 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.068482 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:44.068486 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:44.070188 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260929 03:27:44.070318 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:44.065835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:41515 (local address 127.6.116.190:38281)
0929 03:27:44.065922 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:44.065924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:44.065969 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:27:44.066088 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:44.066089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:44.066102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:44.066170 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:44.066550 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.067098 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.067791 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.067971 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.068022 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.070260 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:41515: BlockingRecv error: recv got EOF from 127.6.116.130:41515 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:44.198539 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:44.193810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:42743)
0929 03:27:44.193959 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:44.193971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:44.194033 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:44.194219 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:44.194220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:44.194392 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:44.194450 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.194452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.195119 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.195120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.195754 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.195757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.196071 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.196072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.196116 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.196673 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:44.196678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:44.198482 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
W20260929 03:27:44.198627 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:44.193982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:42743 (local address 127.6.116.190:38281)
0929 03:27:44.194039 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:44.194040 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:44.194069 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:27:44.194090 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:44.194091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:44.194122 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:44.194189 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:44.194517 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.195057 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.195825 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.196002 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.196053 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.198558 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:42743: BlockingRecv error: recv got EOF from 127.6.116.131:42743 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:44 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:44.953351 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:44.948587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:34077)
0929 03:27:44.948754 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:44.948763 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:44.948809 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:44.948982 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:44.948982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:44.949133 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:44.949181 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.949183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.949836 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.949837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.950408 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:44.950409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.950738 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.950739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:44.950778 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.951419 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:44.951424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:44.953246 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
W20260929 03:27:44.953388 15812 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:44.948713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34077 (local address 127.6.116.190:38281)
0929 03:27:44.948858 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:44.948860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:44.948868 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:27:44.948890 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:44.948890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:44.948909 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:44.948969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:44.949272 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.949767 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.950474 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:44.950670 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:44.950723 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:44.953317 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:34077: BlockingRecv error: recv got EOF from 127.6.116.129:34077 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:45.075276 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:45.070863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:55005)
0929 03:27:45.070976 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:45.070987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:45.071056 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:45.071187 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:45.071188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:45.071330 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:45.071377 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.071379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.072101 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.072102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.072602 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.072604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.072916 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.072916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.072953 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.073366 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:45.073370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:45.075120 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
W20260929 03:27:45.075325 15812 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:45.070922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55005 (local address 127.6.116.190:38281)
0929 03:27:45.070998 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:45.071000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:45.071083 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:27:45.071107 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:45.071108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:45.071121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:45.071181 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:45.071428 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.072038 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.072671 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.072852 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.072906 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.075226 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:55005: BlockingRecv error: recv got EOF from 127.6.116.130:55005 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:45.203325 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:45.199179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:48277)
0929 03:27:45.199335 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:45.199345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:45.199396 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:45.199520 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:45.199520 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:45.199671 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:45.199717 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.199719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.200286 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.200287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.200799 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.200800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.201106 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.201107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.201145 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.201561 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:45.201565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:45.203268 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
W20260929 03:27:45.203498 15812 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:45.199252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:48277 (local address 127.6.116.190:38281)
0929 03:27:45.199310 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:45.199311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:45.199398 (+ 87us) server_negotiation.cc:407] Connection header received
0929 03:27:45.199449 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:45.199449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:45.199462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:45.199500 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:45.199776 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.200229 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.200870 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.201040 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.201089 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.203437 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:48277: BlockingRecv error: recv got EOF from 127.6.116.131:48277 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:45 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:45.958472 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:45.953855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:39345)
0929 03:27:45.953998 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:45.954010 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:45.954079 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:45.954238 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:45.954239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:45.954492 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:45.954555 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.954558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.955268 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.955268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.955771 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:45.955773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.956127 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.956128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:45.956164 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.956633 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:45.956637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:45.958412 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":69,"threads_started":1}
W20260929 03:27:45.958630 15816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:45.953903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:39345 (local address 127.6.116.190:38281)
0929 03:27:45.954083 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:45.954086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:45.954096 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:45.954124 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:45.954125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:45.954138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:45.954211 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:45.954643 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.955185 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.955864 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:45.956060 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:45.956119 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:45.958504 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:39345: BlockingRecv error: recv got EOF from 127.6.116.129:39345 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":82,"threads_started":1}
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:46.080418 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:46.075756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:42457)
0929 03:27:46.075863 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:46.075873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:46.075939 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:46.076042 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:46.076043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:46.076215 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:46.076261 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.076263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.077093 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.077094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.077631 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.077633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.077940 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.077941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.077977 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.078481 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:46.078486 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:46.080304 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
W20260929 03:27:46.080456 15816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:46.075810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:42457 (local address 127.6.116.190:38281)
0929 03:27:46.075872 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:46.075874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:46.075924 (+ 50us) server_negotiation.cc:407] Connection header received
0929 03:27:46.075981 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:46.075981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:46.075991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:46.076040 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:46.076341 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.077022 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.077698 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.077878 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.077929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.080389 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:42457: BlockingRecv error: recv got EOF from 127.6.116.130:42457 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:46.211474 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:46.203843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:33815)
0929 03:27:46.203986 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:46.204005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:46.204090 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:46.204232 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:46.204233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:46.204436 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:46.204488 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.204490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.205159 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.205160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.205682 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.205684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.206025 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.206026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.206088 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.209534 (+ 3446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:46.209540 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:46.211371 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260929 03:27:46.211511 15816 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:46.203915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:33815 (local address 127.6.116.190:38281)
0929 03:27:46.203990 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:46.203992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:46.204071 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:27:46.204161 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:46.204162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:46.204176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:46.204228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:46.204575 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.205056 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.205763 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.205952 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.206026 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.211441 (+ 5415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:33815: BlockingRecv error: recv got EOF from 127.6.116.131:33815 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:46 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:46.963232 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:46.958896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:43285)
0929 03:27:46.959082 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:46.959093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:46.959155 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:46.959285 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:46.959285 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:46.959442 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:46.959492 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.959493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.960117 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.960118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.960638 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:46.960640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.960951 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.960952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:46.960988 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.961428 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:46.961434 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:46.963126 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
W20260929 03:27:46.963277 15820 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:46.959020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43285 (local address 127.6.116.190:38281)
0929 03:27:46.959151 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:46.959153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:46.959186 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:27:46.959206 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:46.959207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:46.959216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:46.959264 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:46.959556 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.960044 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.960703 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:46.960885 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:46.960933 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:46.963191 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:43285: BlockingRecv error: recv got EOF from 127.6.116.129:43285 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:47.085270 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:47.080853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:51857)
0929 03:27:47.080981 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:47.080993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:47.081049 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:47.081177 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:47.081177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:47.081327 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:47.081375 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.081377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.082037 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.082038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.082708 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.082710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.083015 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.083016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.083069 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.083537 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:47.083541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:47.085176 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260929 03:27:47.085320 15820 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:47.080916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:51857 (local address 127.6.116.190:38281)
0929 03:27:47.080979 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:47.080981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:47.081054 (+ 73us) server_negotiation.cc:407] Connection header received
0929 03:27:47.081109 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:47.081110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:47.081120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:47.081160 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:47.081437 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.081966 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.082774 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.082955 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.083007 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.085255 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:51857: BlockingRecv error: recv got EOF from 127.6.116.130:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:47.216441 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:47.211989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:41551)
0929 03:27:47.212174 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:47.212187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:47.212265 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:47.212404 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:47.212405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:47.212581 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:47.212624 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.212625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.213252 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.213253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.213751 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.213753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.214083 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.214084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.214146 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.214638 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:47.214643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:47.216350 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":110,"threads_started":1}
W20260929 03:27:47.216473 15820 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:47.212111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:41551 (local address 127.6.116.190:38281)
0929 03:27:47.212178 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:47.212181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:47.212249 (+ 68us) server_negotiation.cc:407] Connection header received
0929 03:27:47.212327 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:47.212328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:47.212340 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:47.212399 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:47.212703 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.213174 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.213834 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.214013 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.214083 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.216418 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:41551: BlockingRecv error: recv got EOF from 127.6.116.131:41551 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:47 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:47.967978 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:47.963783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:38743)
0929 03:27:47.963913 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:47.963923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:47.963983 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:47.964129 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:47.964129 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:47.964286 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:47.964331 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.964332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.964978 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.964978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.965504 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:47.965505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.965817 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.965817 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:47.965854 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.966261 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:47.966266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:47.967914 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
W20260929 03:27:47.968101 15824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:47.963838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:38743 (local address 127.6.116.190:38281)
0929 03:27:47.963968 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:47.963970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:47.963985 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:27:47.964040 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:47.964040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:47.964050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:47.964109 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:47.964405 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.964916 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.965578 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:47.965757 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:47.965805 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:47.967999 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:38743: BlockingRecv error: recv got EOF from 127.6.116.129:38743 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:48.089977 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:48.085777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:49829)
0929 03:27:48.085891 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:48.085902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:48.085962 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:48.086080 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:48.086081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:48.086256 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:48.086304 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.086306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.086906 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.086907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.087450 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.087452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.087757 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.087758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.087808 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.088201 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:48.088205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:48.089911 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
W20260929 03:27:48.090078 15824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:48.085840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:49829 (local address 127.6.116.190:38281)
0929 03:27:48.085925 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:48.085926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:48.085948 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:27:48.086014 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:48.086015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:48.086024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:48.086073 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:48.086369 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.086844 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.087517 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.087695 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.087746 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.090005 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:49829: BlockingRecv error: recv got EOF from 127.6.116.130:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:48.221542 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:48.216926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:52495)
0929 03:27:48.217106 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:48.217119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:48.217203 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:48.217322 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:48.217323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:48.217534 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:48.217582 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.217584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.218230 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.218232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.218828 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.218830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.219243 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.219243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.219302 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.219712 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:48.219717 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:48.221479 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":99,"threads_started":1}
W20260929 03:27:48.221632 15824 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:48.217014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:52495 (local address 127.6.116.190:38281)
0929 03:27:48.217096 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:48.217099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:48.217182 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:27:48.217255 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:48.217256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:48.217269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:48.217318 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:48.217659 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.218153 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.218919 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.219174 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.219245 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.221567 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:52495: BlockingRecv error: recv got EOF from 127.6.116.131:52495 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:48 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:48.973023 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:48.968639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:37937)
0929 03:27:48.968783 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:48.968794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:48.968873 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:48.969034 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:48.969035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:48.969198 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:48.969243 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.969245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.969867 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.969868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.970367 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:48.970369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.970677 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.970678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:48.970717 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.971220 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:48.971224 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:48.972924 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
W20260929 03:27:48.973084 15828 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:48.968706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:37937 (local address 127.6.116.190:38281)
0929 03:27:48.968803 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:48.968805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:48.968857 (+ 52us) server_negotiation.cc:407] Connection header received
0929 03:27:48.968938 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:48.968939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:48.968950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:48.969016 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:48.969312 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.969806 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.970435 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:48.970617 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:48.970664 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:48.973012 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:37937: BlockingRecv error: recv got EOF from 127.6.116.129:37937 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":58,"threads_started":1}
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:49.095201 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:49.090488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:35519)
0929 03:27:49.090658 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:49.090669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:49.090753 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:49.090902 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:49.090903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:49.091097 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:49.091140 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.091142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.091856 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.091858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.092383 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.092384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.092768 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.092769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.092833 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.093292 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:49.093297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:49.095112 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
W20260929 03:27:49.095247 15828 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:49.090576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:35519 (local address 127.6.116.190:38281)
0929 03:27:49.090644 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:49.090646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:49.090736 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:27:49.090809 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:49.090810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:49.090821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:49.090896 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:49.091216 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.091773 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.092472 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.092669 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.092755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.095185 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:35519: BlockingRecv error: recv got EOF from 127.6.116.130:35519 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:49.226804 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:49.222067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:55743)
0929 03:27:49.222214 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:49.222226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:49.222313 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:49.222455 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:49.222456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:49.222636 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:49.222678 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.222680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.223393 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.223394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.223917 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.223919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.224286 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.224287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.224361 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.224816 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:49.224820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:49.226748 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
W20260929 03:27:49.226899 15828 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:49.222156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:55743 (local address 127.6.116.190:38281)
0929 03:27:49.222224 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:49.222226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:49.222296 (+ 70us) server_negotiation.cc:407] Connection header received
0929 03:27:49.222393 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:49.222394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:49.222405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:49.222450 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:49.222754 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.223318 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.224021 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.224208 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.224286 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.226833 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:55743: BlockingRecv error: recv got EOF from 127.6.116.131:55743 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:49 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:49.978271 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:49.973519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:34881)
0929 03:27:49.973657 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:49.973669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:49.973728 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:49.973931 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:49.973932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:49.974124 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:49.974166 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.974168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.974912 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.974913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.975529 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:49.975531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.975914 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.975915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:49.975972 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.976469 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:49.976487 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:49.978172 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260929 03:27:49.978328 15832 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:49.973613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34881 (local address 127.6.116.190:38281)
0929 03:27:49.973722 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:49.973724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:49.973733 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:49.973850 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:49.973851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:49.973863 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:49.973925 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:49.974247 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.974834 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.975619 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:49.975849 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:49.975904 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:49.978255 (+ 2351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:34881: BlockingRecv error: recv got EOF from 127.6.116.129:34881 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":47,"threads_started":1}
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:50.099687 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:50.095605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:49167)
0929 03:27:50.095725 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:50.095733 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:50.095772 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:50.095898 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:50.095899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:50.096046 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:50.096091 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.096093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.096690 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.096691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.097194 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.097195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.097504 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.097505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.097544 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.097953 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:50.097957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:50.099632 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":54,"threads_started":1}
W20260929 03:27:50.099793 15832 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:50.095705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:49167 (local address 127.6.116.190:38281)
0929 03:27:50.095767 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:50.095769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:50.095804 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:27:50.095825 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:50.095826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:50.095835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:50.095873 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:50.096150 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.096628 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.097259 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.097438 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.097487 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.099720 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:49167: BlockingRecv error: recv got EOF from 127.6.116.130:49167 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:50.232168 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:50.227301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:57439)
0929 03:27:50.227478 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:50.227491 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:50.227557 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:50.227693 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:50.227694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:50.227879 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:50.227923 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.227925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.228660 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.228661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.229209 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.229211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.229582 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.229583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.229640 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.230130 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:50.230134 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:50.232066 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
W20260929 03:27:50.232208 15832 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:50.227391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:57439 (local address 127.6.116.190:38281)
0929 03:27:50.227465 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:50.227467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:50.227539 (+ 72us) server_negotiation.cc:407] Connection header received
0929 03:27:50.227627 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:50.227628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:50.227641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:50.227690 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:50.228003 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.228577 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.229295 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.229506 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.229579 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.232144 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:57439: BlockingRecv error: recv got EOF from 127.6.116.131:57439 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:50 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:50.983175 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:50.978723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:39189)
0929 03:27:50.978860 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:50.978869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:50.978935 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:50.979136 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:50.979137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:50.979293 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:50.979341 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.979342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.979956 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.979956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.980448 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:50.980449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.980744 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.980745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:50.980781 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.981195 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:50.981199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:50.983083 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260929 03:27:50.983228 15836 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:50.978778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:39189 (local address 127.6.116.190:38281)
0929 03:27:50.978914 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:50.978916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:50.978925 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:50.978993 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:50.978994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:50.979005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:50.979081 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:50.979404 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.979896 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.980513 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:50.980688 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:50.980734 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:50.983159 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:39189: BlockingRecv error: recv got EOF from 127.6.116.129:39189 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:51.105077 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:51.100115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:56873)
0929 03:27:51.100313 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:51.100325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:51.100411 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:51.100652 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:51.100653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:51.100852 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:51.100896 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.100898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.101645 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.101646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.102199 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.102201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.102584 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.102585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.102649 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.103169 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:51.103173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:51.104927 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
W20260929 03:27:51.105075 15836 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:51.100210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:56873 (local address 127.6.116.190:38281)
0929 03:27:51.100294 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:51.100297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:51.100393 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:27:51.100467 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:51.100468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:51.100480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:51.100638 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:51.101010 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.101551 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.102287 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.102500 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.102575 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.105018 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:56873: BlockingRecv error: recv got EOF from 127.6.116.130:56873 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.131@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:51.240823 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:51.232698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.131:32811)
0929 03:27:51.232872 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:51.232882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:51.232942 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:51.233070 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:51.233070 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:51.233226 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:51.233272 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.233273 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.233927 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.233928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.234438 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.234439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.234758 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.234758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.234794 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.239159 (+ 4365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:51.239164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:51.240769 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":100,"threads_started":1}
W20260929 03:27:51.240998 15836 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:51.232764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:32811 (local address 127.6.116.190:38281)
0929 03:27:51.232831 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:51.232833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:51.232957 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:27:51.232982 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:51.232983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:51.232994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:51.233042 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:51.233333 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.233861 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.234511 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.234694 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.234747 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.240916 (+ 6169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.131:32811: BlockingRecv error: recv got EOF from 127.6.116.131:32811 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:51 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.129@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:51.988147 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:51.983720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.129:43055)
0929 03:27:51.983848 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:51.983858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:51.983909 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:51.984033 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:51.984033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:51.984201 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:51.984248 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.984250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.984900 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.984900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.985412 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:51.985413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.985730 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.985731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:51.985768 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.986191 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:51.986196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:51.988046 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
W20260929 03:27:51.988193 15840 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:51.983802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43055 (local address 127.6.116.190:38281)
0929 03:27:51.983905 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:51.983907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:51.983914 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:27:51.983959 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:51.983960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:51.983970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:51.984029 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:51.984310 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.984833 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.985481 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:51.985673 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:51.985721 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:51.988124 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.129:43055: BlockingRecv error: recv got EOF from 127.6.116.129:43055 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":47,"threads_started":1}
Sep 29 03:27:52 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
Sep 29 03:27:52 dist-test-slave-z2zg krb5kdc[14781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.116.130@KRBTEST.COM for oryx/127.6.116.190@KRBTEST.COM, Server not found in Kerberos database
I20260929 03:27:52.109944 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.105718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38281 (local address 127.6.116.130:52237)
0929 03:27:52.105846 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:52.105863 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:52.105941 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:52.106099 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:52.106100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:52.106262 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:52.106308 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.106309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.106915 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.106916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.107443 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.107445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.107753 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.107753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.107790 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.108193 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:52.108197 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:52.109883 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.116.190:38281: Server oryx/127.6.116.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
W20260929 03:27:52.110065 15840 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:52.105776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:52237 (local address 127.6.116.190:38281)
0929 03:27:52.105828 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.105830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.105924 (+ 94us) server_negotiation.cc:407] Connection header received
0929 03:27:52.106030 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.106030 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.106042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.106088 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.106368 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.106848 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.107511 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.107689 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.107741 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.109970 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.116.130:52237: BlockingRecv error: recv got EOF from 127.6.116.130:52237 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:52.148178 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15364
I20260929 03:27:52.152649 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15516
I20260929 03:27:52.156880 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15650
I20260929 03:27:52.160842 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15294
2026-09-29T03:27:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17330 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T03:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:52Z Disabled control of system clock
I20260929 03:27:52.187222 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34051
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 03:27:52.263902 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.264060 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.264077 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.265188 15845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:52.265224 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.265237 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.265249 15845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:52.265260 15845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:52.266556 15845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:34051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.266779 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.267082 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:27:52.272117 15845 server_base.cc:1034] running on GCE node
W20260929 03:27:52.272063 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.272092 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.272089 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.272532 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.272799 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.273921 15845 hybrid_clock.cc:648] HybridClock initialized: now 1790652472273907 us; error 27 us; skew 500 ppm
I20260929 03:27:52.275159 15845 webserver.cc:460] Webserver started at http://127.6.116.190:37839/ using document root <none> and password file <none>
I20260929 03:27:52.275396 15845 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.275439 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.275573 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.276499 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "7e695865f2e545898cf2ad0cb10eb076"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "505e896cfd3ae3495d40455438baf3c1"
server_key_iv: "2f06cf32d43fc87da55318c99a40e304"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.276819 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "7e695865f2e545898cf2ad0cb10eb076"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "505e896cfd3ae3495d40455438baf3c1"
server_key_iv: "2f06cf32d43fc87da55318c99a40e304"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.278023 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:52.278692 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.278880 15845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:52.278972 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "7e695865f2e545898cf2ad0cb10eb076"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "505e896cfd3ae3495d40455438baf3c1"
server_key_iv: "2f06cf32d43fc87da55318c99a40e304"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.279079 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.296510 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.296759 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.296902 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.300714 15845 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:34051
I20260929 03:27:52.300766 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:34051 every 8 connection(s)
I20260929 03:27:52.301126 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:52.301838 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:52.304237 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076: Bootstrap starting.
I20260929 03:27:52.304785 15912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:52.305066 15912 log.cc:824] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:52.305771 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076: No bootstrap required, opened a new log
I20260929 03:27:52.307526 15912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } }
I20260929 03:27:52.307655 15912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:52.307680 15912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e695865f2e545898cf2ad0cb10eb076, State: Initialized, Role: FOLLOWER
I20260929 03:27:52.307853 15912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } }
I20260929 03:27:52.307914 15912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:52.307960 15912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:52.308012 15912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:52.308560 15912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } }
I20260929 03:27:52.308691 15912 leader_election.cc:302] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e695865f2e545898cf2ad0cb10eb076; no voters:
I20260929 03:27:52.308871 15912 leader_election.cc:288] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:52.308917 15915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:52.309091 15915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [term 1 LEADER]: Becoming Leader. State: Replica: 7e695865f2e545898cf2ad0cb10eb076, State: Running, Role: LEADER
I20260929 03:27:52.309127 15912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:52.309214 15915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } }
I20260929 03:27:52.309680 15917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e695865f2e545898cf2ad0cb10eb076. Latest consensus state: current_term: 1 leader_uuid: "7e695865f2e545898cf2ad0cb10eb076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } } }
I20260929 03:27:52.309759 15917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:52.309830 15916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e695865f2e545898cf2ad0cb10eb076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e695865f2e545898cf2ad0cb10eb076" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34051 } } }
I20260929 03:27:52.309919 15916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076 [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:52.310186 15924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:52.310627 15924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:52.311352 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15845
I20260929 03:27:52.311448 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:52.311637 6610 external_mini_cluster.cc:1428] Setting key 7a74a346d710c963776a6f7e1290d9eb
I20260929 03:27:52.315434 15924 catalog_manager.cc:1347] Generated new cluster ID: 1fdebf987f55414f8588f62b4c2b64ea
I20260929 03:27:52.315486 15924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:52.323029 15924 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:52.323448 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.312060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.6.116.190:34051)
0929 03:27:52.312222 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.312230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.312261 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:52.312366 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.312371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.312415 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.312513 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.312751 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.313563 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.314739 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.315177 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.315308 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.322661 (+ 7353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:52.322664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:52.322664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:52.322867 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:52.322925 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:52.322940 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:52.323007 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:52.323175 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:27:52.323253 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260929 03:27:52.323742 15924 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:52.328939 15924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e695865f2e545898cf2ad0cb10eb076: Generated new TSK 0
I20260929 03:27:52.329102 15924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:52.332538 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--builtin_ntp_servers=127.6.116.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:27:52.412467 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.412680 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.412714 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.413820 15936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.413880 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.413923 15936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:52.415287 15936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.415539 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.415808 15936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:52.421478 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.421474 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.421521 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.421957 15936 server_base.cc:1034] running on GCE node
I20260929 03:27:52.422181 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.422444 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.423622 15936 hybrid_clock.cc:648] HybridClock initialized: now 1790652472423609 us; error 36 us; skew 500 ppm
I20260929 03:27:52.424925 15936 webserver.cc:460] Webserver started at http://127.6.116.129:39585/ using document root <none> and password file <none>
I20260929 03:27:52.425192 15936 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.425256 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.425385 15936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.426694 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "939bc589885048b3aee0d7918cabc09b"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "1a33c4ced4e5514996c04af358e3aee7"
server_key_iv: "2a81d7731bd230b460af7c9e13319388"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.427109 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "939bc589885048b3aee0d7918cabc09b"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "1a33c4ced4e5514996c04af358e3aee7"
server_key_iv: "2a81d7731bd230b460af7c9e13319388"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.428622 15936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:52.429369 15950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.429543 15936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:52.429623 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "939bc589885048b3aee0d7918cabc09b"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "1a33c4ced4e5514996c04af358e3aee7"
server_key_iv: "2a81d7731bd230b460af7c9e13319388"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.429683 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.441900 15936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.442180 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.442292 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.442476 15936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:52.442757 15936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:52.442792 15936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.442817 15936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:52.442840 15936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.449389 15936 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:44373
I20260929 03:27:52.449422 16063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:44373 every 8 connection(s)
I20260929 03:27:52.449744 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:52.454156 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.450386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:57647 (local address 127.6.116.190:34051)
0929 03:27:52.450453 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.450455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.450675 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:27:52.451136 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.451138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.451157 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.451211 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.451757 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.452208 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.453076 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.453248 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.453298 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.453966 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:52.453968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:52.453968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:52.453973 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:52.453985 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:52.453988 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:52.454032 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:52.454085 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 03:27:52.454104 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:52.454319 16066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.450356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34051 (local address 127.6.116.129:57647)
0929 03:27:52.450520 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:52.450571 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:52.450994 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:52.451257 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:52.451260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:52.451268 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:52.451688 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.451692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.452279 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.452280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.453006 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.453008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.453314 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.453315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.453411 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.453849 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:52.453850 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:52.453873 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:52.453893 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:52.454034 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:52.454036 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:52.454038 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:52.454079 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 03:27:52.454147 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260929 03:27:52.454805 16064 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34051
I20260929 03:27:52.454874 16064 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:52.455086 16064 heartbeater.cc:511] Master 127.6.116.190:34051 requested a full tablet report, sending...
I20260929 03:27:52.455519 15876 ts_manager.cc:194] Registered new tserver with Master: 939bc589885048b3aee0d7918cabc09b (127.6.116.129:44373)
I20260929 03:27:52.456037 15876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.129:57647
I20260929 03:27:52.457669 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 15936
I20260929 03:27:52.457738 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:52.457873 6610 external_mini_cluster.cc:1428] Setting key 3019eee4fecf7b63bcea60d972c984cd
I20260929 03:27:52.458671 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--builtin_ntp_servers=127.6.116.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:27:52.534654 16067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.534852 16067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.534873 16067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.535943 16067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.535979 16067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.536001 16067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:52.537242 16067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.537401 16067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.537670 16067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:52.542182 16073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.542253 16072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.542207 16075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.542490 16067 server_base.cc:1034] running on GCE node
I20260929 03:27:52.542726 16067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.542949 16067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.544103 16067 hybrid_clock.cc:648] HybridClock initialized: now 1790652472544087 us; error 27 us; skew 500 ppm
I20260929 03:27:52.545251 16067 webserver.cc:460] Webserver started at http://127.6.116.130:33533/ using document root <none> and password file <none>
I20260929 03:27:52.545478 16067 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.545544 16067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.545684 16067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.546677 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "d9c922c2752047b7b075930b2f839d6d"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "b7b495077baba00f70d02b01000ee54b"
server_key_iv: "c47db5a4050d001354ff21a32eaa0749"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.547020 16067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "d9c922c2752047b7b075930b2f839d6d"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "b7b495077baba00f70d02b01000ee54b"
server_key_iv: "c47db5a4050d001354ff21a32eaa0749"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.548243 16067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:52.548928 16081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.549086 16067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:52.549176 16067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "d9c922c2752047b7b075930b2f839d6d"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "b7b495077baba00f70d02b01000ee54b"
server_key_iv: "c47db5a4050d001354ff21a32eaa0749"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.549256 16067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.560286 16067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.560534 16067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.560678 16067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.560877 16067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:52.561184 16067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:52.561240 16067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.561278 16067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:52.561305 16067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.566622 16067 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:38947
I20260929 03:27:52.566689 16194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:38947 every 8 connection(s)
I20260929 03:27:52.566980 16067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:52.571332 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.567678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:36451 (local address 127.6.116.190:34051)
0929 03:27:52.567717 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.567720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.567922 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:27:52.568339 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.568339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.568348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.568400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.568944 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.569390 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.570235 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.570402 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.570447 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.571156 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:52.571157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:52.571157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:52.571160 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:52.571173 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:52.571176 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:52.571216 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:52.571260 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 03:27:52.571279 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:27:52.571543 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.567641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34051 (local address 127.6.116.130:36451)
0929 03:27:52.567805 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:52.567841 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:52.568265 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:52.568443 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:52.568446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:52.568453 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:52.568883 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.568886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.569459 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.569459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.570169 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.570170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.570465 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.570466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.570576 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.570974 (+ 398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:52.570975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:52.570996 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:52.571015 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:52.571217 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:52.571219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:52.571221 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:52.571262 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 03:27:52.571328 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260929 03:27:52.572021 16195 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34051
I20260929 03:27:52.572096 16195 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:52.572211 16195 heartbeater.cc:511] Master 127.6.116.190:34051 requested a full tablet report, sending...
I20260929 03:27:52.572485 15876 ts_manager.cc:194] Registered new tserver with Master: d9c922c2752047b7b075930b2f839d6d (127.6.116.130:38947)
I20260929 03:27:52.572745 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16067
I20260929 03:27:52.572813 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:52.572908 15876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.130:36451
I20260929 03:27:52.573026 6610 external_mini_cluster.cc:1428] Setting key 9d9ebf2d51818a255afa012b2a24cf61
I20260929 03:27:52.573837 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--builtin_ntp_servers=127.6.116.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 03:27:52.650518 16198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.650735 16198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.650786 16198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.652058 16198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.652113 16198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.652153 16198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:52.653535 16198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.653771 16198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.654023 16198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:52.658550 16206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.658545 16204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.658591 16203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.658684 16198 server_base.cc:1034] running on GCE node
I20260929 03:27:52.658981 16198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.659229 16198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.660384 16198 hybrid_clock.cc:648] HybridClock initialized: now 1790652472660379 us; error 33 us; skew 500 ppm
I20260929 03:27:52.661469 16198 webserver.cc:460] Webserver started at http://127.6.116.131:33863/ using document root <none> and password file <none>
I20260929 03:27:52.661695 16198 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.661758 16198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.661882 16198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.662853 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "fd24c3ee86ef46f4b06443188a907cfb"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "e9aa5fe80b66f742492fdb0bd6dc2f7c"
server_key_iv: "13c17caae8fd15aa7674ee46574b2e20"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.663224 16198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "fd24c3ee86ef46f4b06443188a907cfb"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "e9aa5fe80b66f742492fdb0bd6dc2f7c"
server_key_iv: "13c17caae8fd15aa7674ee46574b2e20"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.664422 16198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:52.665061 16212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.665221 16198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:52.665314 16198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "fd24c3ee86ef46f4b06443188a907cfb"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "e9aa5fe80b66f742492fdb0bd6dc2f7c"
server_key_iv: "13c17caae8fd15aa7674ee46574b2e20"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.665395 16198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.685534 16198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.685777 16198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.685911 16198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.686107 16198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:52.686416 16198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:52.686470 16198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.686506 16198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:52.686532 16198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.691747 16198 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:33359
I20260929 03:27:52.691823 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:33359 every 8 connection(s)
I20260929 03:27:52.692106 16198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:52.696501 15935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.692768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:36419 (local address 127.6.116.190:34051)
0929 03:27:52.692808 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.692810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.692981 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:27:52.693407 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.693408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.693416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.693467 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.694025 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.694490 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.695428 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.695615 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.695666 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.696283 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:52.696284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:52.696284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:52.696289 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:52.696300 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:52.696303 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:52.696345 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:52.696423 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 03:27:52.696446 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:52.696684 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.692724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34051 (local address 127.6.116.131:36419)
0929 03:27:52.692886 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:52.692919 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:52.693314 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:52.693512 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:52.693515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:52.693522 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:52.693960 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.693963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.694562 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.694563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.695356 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:52.695359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:52.695682 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:52.695683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:52.695776 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:52.696169 (+ 393us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:52.696169 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:52.696192 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:52.696210 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:52.696356 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:52.696360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:52.696362 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:52.696410 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:27:52.696477 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260929 03:27:52.697194 16326 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34051
I20260929 03:27:52.697259 16326 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:52.697427 16326 heartbeater.cc:511] Master 127.6.116.190:34051 requested a full tablet report, sending...
I20260929 03:27:52.697795 15876 ts_manager.cc:194] Registered new tserver with Master: fd24c3ee86ef46f4b06443188a907cfb (127.6.116.131:33359)
I20260929 03:27:52.698179 15876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.131:36419
I20260929 03:27:52.698194 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16198
I20260929 03:27:52.698282 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:52.698472 6610 external_mini_cluster.cc:1428] Setting key c38075c2214cdd686305f121fcf60556
I20260929 03:27:52.699359 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:27:52.701025 15935 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:52.700592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.6.116.190:34051)
0929 03:27:52.700646 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.700648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.700726 (+ 78us) server_negotiation.cc:407] Connection header received
0929 03:27:52.700787 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.700788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.700798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.700839 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.700967 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38614: BlockingRecv error: recv got EOF from 127.0.0.1:38614 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:27:52.701102 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15936
I20260929 03:27:52.706194 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16067
I20260929 03:27:52.710448 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16198
I20260929 03:27:52.714306 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 15845
2026-09-29T03:27:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T03:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:52Z Disabled control of system clock
I20260929 03:27:52.740468 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39071
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:39071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:27:52.817667 16337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.817849 16337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.817870 16337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.819001 16337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:52.819065 16337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.819080 16337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.819118 16337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:52.819146 16337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:52.820470 16337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:39071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.820693 16337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.820950 16337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:52.825629 16345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.825716 16342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.825624 16343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.825811 16337 server_base.cc:1034] running on GCE node
I20260929 03:27:52.826030 16337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.826339 16337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.827497 16337 hybrid_clock.cc:648] HybridClock initialized: now 1790652472827475 us; error 33 us; skew 500 ppm
I20260929 03:27:52.828770 16337 webserver.cc:460] Webserver started at http://127.6.116.190:40653/ using document root <none> and password file <none>
I20260929 03:27:52.829008 16337 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.829074 16337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.829365 16337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.830370 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "4a9a49bd16b94288b5cbb7296c19507e"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "c333605f13e1fd9d4e5c61ee645af1a7"
server_key_iv: "f4e4b5f73d08c3093f67e6d1bc836055"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.830762 16337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "4a9a49bd16b94288b5cbb7296c19507e"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "c333605f13e1fd9d4e5c61ee645af1a7"
server_key_iv: "f4e4b5f73d08c3093f67e6d1bc836055"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.832078 16337 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:52.833686 16351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.833909 16337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:52.833999 16337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "4a9a49bd16b94288b5cbb7296c19507e"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "c333605f13e1fd9d4e5c61ee645af1a7"
server_key_iv: "f4e4b5f73d08c3093f67e6d1bc836055"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.834075 16337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.837226 16337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.837469 16337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.837615 16337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.841275 16337 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:39071
I20260929 03:27:52.841302 16403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:39071 every 8 connection(s)
I20260929 03:27:52.841622 16337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:52.842237 16404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:52.844110 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16337
I20260929 03:27:52.844208 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:52.844374 6610 external_mini_cluster.cc:1428] Setting key e9194a7539cbd7b764764bc44e70db8d
I20260929 03:27:52.845167 16404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e: Bootstrap starting.
I20260929 03:27:52.845808 16404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:52.846148 16404 log.cc:824] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:52.846891 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:52.844848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.6.116.190:39071)
0929 03:27:52.845050 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:52.845058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:52.845089 (+ 31us) server_negotiation.cc:407] Connection header received
0929 03:27:52.845627 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:52.845635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:52.845684 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:52.845836 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:52.846474 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:52.846477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:52.846478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:52.846483 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:52.846531 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:52.846541 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:52.846626 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:52.846695 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 03:27:52.846748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260929 03:27:52.846911 16404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e: No bootstrap required, opened a new log
I20260929 03:27:52.848434 16404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } }
I20260929 03:27:52.848572 16404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:52.848595 16404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9a49bd16b94288b5cbb7296c19507e, State: Initialized, Role: FOLLOWER
I20260929 03:27:52.848738 16404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } }
I20260929 03:27:52.848821 16404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:52.848867 16404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:52.848927 16404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:52.849706 16404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } }
I20260929 03:27:52.849828 16404 leader_election.cc:302] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a9a49bd16b94288b5cbb7296c19507e; no voters:
I20260929 03:27:52.849977 16404 leader_election.cc:288] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:52.850003 16409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:52.850193 16409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [term 1 LEADER]: Becoming Leader. State: Replica: 4a9a49bd16b94288b5cbb7296c19507e, State: Running, Role: LEADER
I20260929 03:27:52.850258 16404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:52.850289 16409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } }
I20260929 03:27:52.850652 16411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9a49bd16b94288b5cbb7296c19507e. Latest consensus state: current_term: 1 leader_uuid: "4a9a49bd16b94288b5cbb7296c19507e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } } }
I20260929 03:27:52.850983 16411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:52.850900 16410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a9a49bd16b94288b5cbb7296c19507e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9a49bd16b94288b5cbb7296c19507e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39071 } } }
I20260929 03:27:52.851163 16410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:52.851207 16418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:52.851646 16418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:52.852999 16418 catalog_manager.cc:1347] Generated new cluster ID: 6c3b56af155e41858ab8ea0b48be2217
I20260929 03:27:52.853065 16418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:52.871387 16418 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:27:52.871531 16418 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:52.876008 16418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a9a49bd16b94288b5cbb7296c19507e: Generated new TSK 0
I20260929 03:27:52.876179 16418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:27:52.882501 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--builtin_ntp_servers=127.6.116.148:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:27:52.978590 16428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:52.978796 16428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:52.978832 16428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:52.979985 16428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:52.980046 16428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:52.980095 16428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:52.981406 16428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:52.981652 16428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:52.981917 16428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:52.987805 16434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.987811 16436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:52.987828 16433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:52.988135 16428 server_base.cc:1034] running on GCE node
I20260929 03:27:52.988333 16428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:52.988566 16428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:52.989696 16428 hybrid_clock.cc:648] HybridClock initialized: now 1790652472989675 us; error 37 us; skew 500 ppm
I20260929 03:27:52.990959 16428 webserver.cc:460] Webserver started at http://127.6.116.129:36655/ using document root <none> and password file <none>
I20260929 03:27:52.991211 16428 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:52.991277 16428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:52.991405 16428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:52.992369 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "e08958f5dabe4052b15be227d65ea5d7"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "724385fd02ef0ac95806ffaed40d075c"
server_key_iv: "7c6313f49a745e5b9a7194dd419790e9"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.992733 16428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "e08958f5dabe4052b15be227d65ea5d7"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "724385fd02ef0ac95806ffaed40d075c"
server_key_iv: "7c6313f49a745e5b9a7194dd419790e9"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.993994 16428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:27:52.994693 16442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.994867 16428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:52.994961 16428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "e08958f5dabe4052b15be227d65ea5d7"
format_stamp: "Formatted at 2026-09-29 03:27:52 on dist-test-slave-z2zg"
server_key: "724385fd02ef0ac95806ffaed40d075c"
server_key_iv: "7c6313f49a745e5b9a7194dd419790e9"
server_key_version: "encryptionkey@0"
I20260929 03:27:52.995102 16428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:52.998250 16428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:52.998487 16428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:52.998618 16428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:52.998821 16428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:52.999192 16428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:52.999243 16428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:52.999282 16428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:52.999308 16428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.004830 16428 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:34403
I20260929 03:27:53.004874 16555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:34403 every 8 connection(s)
I20260929 03:27:53.005148 16428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:53.007431 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16428
I20260929 03:27:53.007506 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:53.007704 6610 external_mini_cluster.cc:1428] Setting key 5869afd728c520e3722cd584fe272d76
I20260929 03:27:53.007890 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.005704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:41459 (local address 127.6.116.190:39071)
0929 03:27:53.005763 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:53.005765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:53.005967 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:27:53.006386 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:53.006387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:53.006393 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:53.006440 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:53.007628 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:53.007629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:53.007630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:53.007634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:53.007649 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:53.007652 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:53.007704 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:53.007807 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:27:53.007823 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:27:53.008074 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.005712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39071 (local address 127.6.116.129:41459)
0929 03:27:53.005867 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:53.005895 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:53.006306 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:53.006484 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:53.006487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:53.006494 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:53.007455 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:53.007456 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:53.007503 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:53.007536 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:53.007710 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:53.007714 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:53.007717 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:53.007813 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 03:27:53.007870 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260929 03:27:53.008523 16556 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39071
I20260929 03:27:53.008601 16556 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:53.008740 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--builtin_ntp_servers=127.6.116.148:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 03:27:53.008754 16556 heartbeater.cc:511] Master 127.6.116.190:39071 requested a full tablet report, sending...
I20260929 03:27:53.009224 16368 ts_manager.cc:194] Registered new tserver with Master: e08958f5dabe4052b15be227d65ea5d7 (127.6.116.129:34403)
W20260929 03:27:53.085251 16559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:53.085420 16559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:53.085440 16559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:53.086563 16559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:53.086597 16559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:53.086622 16559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:53.087858 16559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:53.088013 16559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:53.088344 16559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:53.092801 16567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:53.093000 16564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:53.093045 16559 server_base.cc:1034] running on GCE node
W20260929 03:27:53.093099 16565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:53.093289 16559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:53.093580 16559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:53.094734 16559 hybrid_clock.cc:648] HybridClock initialized: now 1790652473094729 us; error 35 us; skew 500 ppm
I20260929 03:27:53.095894 16559 webserver.cc:460] Webserver started at http://127.6.116.130:39791/ using document root <none> and password file <none>
I20260929 03:27:53.096112 16559 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:53.096177 16559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:53.096316 16559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:53.097328 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "f195a2c19f8f4403aef5802a5303300c"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "c3ad2b1f51e9caf36dd705323e729969"
server_key_iv: "71c7213193f6430d39b51de1e7c0b967"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.097725 16559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "f195a2c19f8f4403aef5802a5303300c"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "c3ad2b1f51e9caf36dd705323e729969"
server_key_iv: "71c7213193f6430d39b51de1e7c0b967"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.099028 16559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:27:53.099803 16573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.099965 16559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:53.100054 16559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "f195a2c19f8f4403aef5802a5303300c"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "c3ad2b1f51e9caf36dd705323e729969"
server_key_iv: "71c7213193f6430d39b51de1e7c0b967"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.100132 16559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:53.103288 16559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:53.103529 16559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:53.103681 16559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:53.103873 16559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:53.104146 16559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:53.104197 16559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.104231 16559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:53.104256 16559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.109771 16559 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:40837
I20260929 03:27:53.109817 16686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:40837 every 8 connection(s)
I20260929 03:27:53.110083 16559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:53.112783 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.110724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:52897 (local address 127.6.116.190:39071)
0929 03:27:53.110787 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:53.110789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:53.110972 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:27:53.111407 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:53.111408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:53.111413 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:53.111494 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:53.112605 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:53.112607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:53.112607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:53.112611 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:53.112625 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:53.112629 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:53.112682 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:53.112726 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 03:27:53.112737 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:27:53.112948 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.110700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39071 (local address 127.6.116.130:52897)
0929 03:27:53.110858 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:53.110893 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:53.111321 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:53.111539 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:53.111542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:53.111550 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:53.112468 (+ 918us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:53.112469 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:53.112493 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:53.112516 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:53.112683 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:53.112685 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:53.112687 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:27:53.112732 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:27:53.112771 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":78,"threads_started":1}
I20260929 03:27:53.113243 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16559
I20260929 03:27:53.113359 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:53.113423 16687 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39071
I20260929 03:27:53.113503 16687 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:53.113566 6610 external_mini_cluster.cc:1428] Setting key e98701357bc3e0d947fd2f181458b343
I20260929 03:27:53.113600 16687 heartbeater.cc:511] Master 127.6.116.190:39071 requested a full tablet report, sending...
I20260929 03:27:53.113919 16367 ts_manager.cc:194] Registered new tserver with Master: f195a2c19f8f4403aef5802a5303300c (127.6.116.130:40837)
I20260929 03:27:53.114421 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--builtin_ntp_servers=127.6.116.148:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 03:27:53.190918 16690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:53.191145 16690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:53.191179 16690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:53.192301 16690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:53.192363 16690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:53.192404 16690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:53.193845 16690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:53.194096 16690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:53.194355 16690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:53.198949 16695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:53.198967 16698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:53.198968 16696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:53.199179 16690 server_base.cc:1034] running on GCE node
I20260929 03:27:53.199406 16690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:53.199628 16690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:53.200786 16690 hybrid_clock.cc:648] HybridClock initialized: now 1790652473200775 us; error 24 us; skew 500 ppm
I20260929 03:27:53.201923 16690 webserver.cc:460] Webserver started at http://127.6.116.131:35015/ using document root <none> and password file <none>
I20260929 03:27:53.202168 16690 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:53.202234 16690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:53.202375 16690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:53.203459 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "c054cc00bf0c40eaa06e54d3f431ceb8"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "3b320e7a85b129263131113f4a14534e"
server_key_iv: "0666ad894ec22884092a376d29a44506"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.203850 16690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "c054cc00bf0c40eaa06e54d3f431ceb8"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "3b320e7a85b129263131113f4a14534e"
server_key_iv: "0666ad894ec22884092a376d29a44506"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.205118 16690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:27:53.205845 16704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.206028 16690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:53.206125 16690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "c054cc00bf0c40eaa06e54d3f431ceb8"
format_stamp: "Formatted at 2026-09-29 03:27:53 on dist-test-slave-z2zg"
server_key: "3b320e7a85b129263131113f4a14534e"
server_key_iv: "0666ad894ec22884092a376d29a44506"
server_key_version: "encryptionkey@0"
I20260929 03:27:53.206234 16690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:53.209440 16690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:53.209686 16690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:53.209820 16690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:53.210027 16690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:53.210304 16690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:53.210356 16690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.210393 16690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:53.210422 16690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:53.215968 16690 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:37291
I20260929 03:27:53.216010 16817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:37291 every 8 connection(s)
I20260929 03:27:53.216276 16690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:27:53.218433 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16690
I20260929 03:27:53.218518 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:53.218707 6610 external_mini_cluster.cc:1428] Setting key 11182450af9b030c1b1b3b15603e7964
I20260929 03:27:53.218863 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.216853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:46155 (local address 127.6.116.190:39071)
0929 03:27:53.216896 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:53.216899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:53.217111 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:27:53.217541 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:53.217543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:53.217550 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:53.217607 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:53.218641 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:53.218643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:53.218644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:27:53.218649 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:53.218664 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:27:53.218669 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:53.218717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:53.218784 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 03:27:53.218799 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:27:53.219067 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:53.216837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39071 (local address 127.6.116.131:46155)
0929 03:27:53.217000 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:53.217028 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:53.217436 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:53.217666 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:53.217669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:53.217678 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:53.218504 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:27:53.218505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:53.218538 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:27:53.218568 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:53.218714 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:53.218717 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:53.218721 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:27:53.218791 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:27:53.218845 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260929 03:27:53.219535 16818 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39071
I20260929 03:27:53.219621 16818 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:53.219751 16818 heartbeater.cc:511] Master 127.6.116.190:39071 requested a full tablet report, sending...
I20260929 03:27:53.220078 16367 ts_manager.cc:194] Registered new tserver with Master: c054cc00bf0c40eaa06e54d3f431ceb8 (127.6.116.131:37291)
I20260929 03:27:53.230388 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 03:27:53.232106 16407 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:27:53.231704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.6.116.190:39071)
0929 03:27:53.231765 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:53.231767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:53.231872 (+ 105us) server_negotiation.cc:407] Connection header received
0929 03:27:53.231917 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:53.231918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:53.231921 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:53.231969 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:53.232067 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52250: BlockingRecv error: recv got EOF from 127.0.0.1:52250 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 03:27:53.232246 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16428
I20260929 03:27:53.237046 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16559
I20260929 03:27:53.240918 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16690
I20260929 03:27:53.244819 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16337
2026-09-29T03:27:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:53 dist-test-slave-z2zg krb5kdc[16828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:53 dist-test-slave-z2zg krb5kdc[16828](info): set up 2 sockets
Sep 29 03:27:53 dist-test-slave-z2zg krb5kdc[16828](info): commencing operation
krb5kdc: starting...
W20260929 03:27:54.269778 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:27:54.388044 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45373
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:45373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:54.468912 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:54.469172 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:54.469214 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:54.470378 16841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:27:54.470448 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:54.470471 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:54.470492 16841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:27:54.470523 16841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:27:54.472074 16841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:45373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:54.472347 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:54.472568 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:54.477299 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.477281 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.477512 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:54.477515 16841 server_base.cc:1034] running on GCE node
I20260929 03:27:54.477900 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:54.478192 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:54.479352 16841 hybrid_clock.cc:648] HybridClock initialized: now 1790652474479339 us; error 61 us; skew 500 ppm
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:54.481922 16841 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:27:54.482328 16841 webserver.cc:460] Webserver started at http://127.6.116.190:35725/ using document root <none> and password file <none>
I20260929 03:27:54.482547 16841 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:54.482640 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:54.482780 16841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:54.483932 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "e13bb9b564d5431b950cfd67964c8c6b"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "85bfc6fa9c534576e64fedeb6db9201d"
server_key_iv: "502d667c6208c1fd30ab618c5de01321"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.484292 16841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "e13bb9b564d5431b950cfd67964c8c6b"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "85bfc6fa9c534576e64fedeb6db9201d"
server_key_iv: "502d667c6208c1fd30ab618c5de01321"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.485608 16841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:54.486328 16856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.486509 16841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:27:54.486624 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "e13bb9b564d5431b950cfd67964c8c6b"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "85bfc6fa9c534576e64fedeb6db9201d"
server_key_iv: "502d667c6208c1fd30ab618c5de01321"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.486698 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:54.496440 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:54.498741 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:54.498901 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:54.502880 16841 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:45373
I20260929 03:27:54.502920 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:45373 every 8 connection(s)
I20260929 03:27:54.503319 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:27:54.504324 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:54.506690 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: Bootstrap starting.
I20260929 03:27:54.507355 16909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:54.507663 16909 log.cc:824] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:54.508355 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: No bootstrap required, opened a new log
I20260929 03:27:54.509999 16909 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } }
I20260929 03:27:54.510135 16909 raft_consensus.cc:374] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:54.510188 16909 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13bb9b564d5431b950cfd67964c8c6b, State: Initialized, Role: FOLLOWER
I20260929 03:27:54.510356 16909 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } }
I20260929 03:27:54.510416 16909 raft_consensus.cc:388] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:27:54.510437 16909 raft_consensus.cc:482] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:27:54.510511 16909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:54.511335 16909 raft_consensus.cc:504] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } }
I20260929 03:27:54.511446 16909 leader_election.cc:302] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e13bb9b564d5431b950cfd67964c8c6b; no voters:
I20260929 03:27:54.511637 16909 leader_election.cc:288] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:27:54.511691 16912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:54.511853 16912 raft_consensus.cc:686] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [term 1 LEADER]: Becoming Leader. State: Replica: e13bb9b564d5431b950cfd67964c8c6b, State: Running, Role: LEADER
I20260929 03:27:54.511909 16909 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:27:54.511935 16912 consensus_queue.cc:237] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } }
I20260929 03:27:54.512286 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e13bb9b564d5431b950cfd67964c8c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } } }
I20260929 03:27:54.512377 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:54.512282 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e13bb9b564d5431b950cfd67964c8c6b. Latest consensus state: current_term: 1 leader_uuid: "e13bb9b564d5431b950cfd67964c8c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e13bb9b564d5431b950cfd67964c8c6b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 45373 } } }
I20260929 03:27:54.512429 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b [sys.catalog]: This master's current role is: LEADER
I20260929 03:27:54.512645 16920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:27:54.512770 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16841
I20260929 03:27:54.512867 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:27:54.513119 6610 external_mini_cluster.cc:1428] Setting key af95ecd0b6796f5ccc65c7c147930a37
I20260929 03:27:54.513166 16920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:27:54.514981 16920 catalog_manager.cc:1347] Generated new cluster ID: d682bf66c18b45b1a97cfb4047b62ce6
I20260929 03:27:54.515034 16920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:27:54.519572 16920 catalog_manager.cc:1370] Generated new certificate authority record
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:54.520228 16920 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:27:54.523627 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:54.513789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.6.116.190:45373)
0929 03:27:54.514077 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:54.514087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:54.514121 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:27:54.514231 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:54.514238 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:54.514293 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:54.514435 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:54.515310 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.516224 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.516971 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.517425 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.517550 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:54.520172 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:54.520175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:54.520176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:54.520188 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:54.522059 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.522446 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.522450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.522450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.522552 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.522762 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.522763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.522763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.522995 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:54.523207 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:54.523273 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:27:54.523429 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":73,"threads_started":1}
I20260929 03:27:54.526500 16920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: Generated new TSK 0
I20260929 03:27:54.526688 16920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:27:54.558125 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:54.652073 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:54.652266 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:54.652287 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:54.653446 16935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:54.653486 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:54.653512 16935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:27:54.654883 16935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:54.655143 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:54.655406 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:27:54.660303 16935 server_base.cc:1034] running on GCE node
W20260929 03:27:54.660261 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.660259 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.660486 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:54.660713 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:54.660979 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:54.662142 16935 hybrid_clock.cc:648] HybridClock initialized: now 1790652474662124 us; error 34 us; skew 500 ppm
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:54.664716 16935 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:27:54.665099 16935 webserver.cc:460] Webserver started at http://127.6.116.129:38195/ using document root <none> and password file <none>
I20260929 03:27:54.665287 16935 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:54.665323 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:54.665405 16935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:54.666368 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "7e90d8dca90d4834988cf150840991f0"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "c5c0d5b962b2a507b91585a5215ab2ca"
server_key_iv: "364fa1aa07c6f1155319636e5872db9a"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.666720 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "7e90d8dca90d4834988cf150840991f0"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "c5c0d5b962b2a507b91585a5215ab2ca"
server_key_iv: "364fa1aa07c6f1155319636e5872db9a"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.668092 16935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 03:27:54.668783 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.668989 16935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:27:54.669071 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "7e90d8dca90d4834988cf150840991f0"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "c5c0d5b962b2a507b91585a5215ab2ca"
server_key_iv: "364fa1aa07c6f1155319636e5872db9a"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.669126 16935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:54.693900 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:54.696316 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:54.696427 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:54.696597 16935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:54.696928 16935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:54.696975 16935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.696998 16935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:54.697047 16935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.702895 16935 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:40777
I20260929 03:27:54.702984 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:40777 every 8 connection(s)
I20260929 03:27:54.703351 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:27:54.703800 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 16935
I20260929 03:27:54.703920 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:27:54.704131 6610 external_mini_cluster.cc:1428] Setting key efeaff9348988f2d933faf8f0b7098e0
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:27:54.713954 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:54.704215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:55551 (local address 127.6.116.190:45373)
0929 03:27:54.704268 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:54.704271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:54.704512 (+ 241us) server_negotiation.cc:407] Connection header received
0929 03:27:54.705184 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:54.705185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:54.705194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:54.705233 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:54.706556 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.707144 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.708162 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.708400 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.708492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:54.710869 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:54.710870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:54.710870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:54.710877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:54.712410 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.712876 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.712877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.712878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.712963 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.713248 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.713249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.713249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.713371 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:54.713457 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:54.713767 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:27:54.713860 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:27:54.714148 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:54.704163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45373 (local address 127.6.116.129:55551)
0929 03:27:54.704333 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:54.704403 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:54.705084 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:54.705311 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:54.705316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:54.706171 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:54.706466 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:54.706473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.707238 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.707239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:54.708055 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:54.708058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.708503 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.708504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:54.708657 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:54.709164 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:54.709172 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:54.710761 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:54.712506 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:54.712514 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:54.712516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:54.712791 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:54.713039 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:54.713040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:54.713040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:54.713151 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:54.713451 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:54.713458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:54.713617 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 03:27:54.713744 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 03:27:54.713897 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260929 03:27:54.714705 17064 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45373
I20260929 03:27:54.714784 17064 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:54.714991 17064 heartbeater.cc:511] Master 127.6.116.190:45373 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
I20260929 03:27:54.715586 16873 ts_manager.cc:194] Registered new tserver with Master: 7e90d8dca90d4834988cf150840991f0 (127.6.116.129:40777)
I20260929 03:27:54.716199 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:55551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:27:54.729295 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:54.813024 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:54.813277 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:54.813331 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:54.814502 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:54.814548 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:54.814576 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:27:54.815970 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:54.816165 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:54.816355 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:54.821535 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.821513 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.821556 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:54.821832 17069 server_base.cc:1034] running on GCE node
I20260929 03:27:54.822080 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:54.822333 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:54.823498 17069 hybrid_clock.cc:648] HybridClock initialized: now 1790652474823472 us; error 55 us; skew 500 ppm
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:54.826009 17069 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:27:54.826435 17069 webserver.cc:460] Webserver started at http://127.6.116.130:33831/ using document root <none> and password file <none>
I20260929 03:27:54.826633 17069 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:54.826673 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:54.826761 17069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:54.827824 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "e01a8189e92f4977902aa0a093411574"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "539968076cc4c96f72ed7b1d646c397f"
server_key_iv: "d63e4431419d1b3ed65f42f530da3088"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.828147 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "e01a8189e92f4977902aa0a093411574"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "539968076cc4c96f72ed7b1d646c397f"
server_key_iv: "d63e4431419d1b3ed65f42f530da3088"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.829701 17069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:27:54.830417 17084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.830621 17069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:27:54.830706 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "e01a8189e92f4977902aa0a093411574"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "539968076cc4c96f72ed7b1d646c397f"
server_key_iv: "d63e4431419d1b3ed65f42f530da3088"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.830760 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:54.843349 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:54.845304 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:54.845443 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:54.845679 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:54.845997 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:54.846031 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.846053 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:54.846097 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.851758 17069 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:44121
I20260929 03:27:54.851835 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:44121 every 8 connection(s)
I20260929 03:27:54.852133 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:27:54.854014 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 17069
I20260929 03:27:54.854125 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:27:54.854333 6610 external_mini_cluster.cc:1428] Setting key 79b3422d46eee34558c751374e461355
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:54.862274 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:54.852832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:40489 (local address 127.6.116.190:45373)
0929 03:27:54.852912 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:54.852914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:54.853090 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:27:54.853582 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:54.853583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:54.853593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:54.853646 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:54.854807 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.855404 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.856793 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.857004 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.857089 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:54.859532 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:54.859533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:54.859533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:54.859539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:54.860886 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.861358 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.861359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.861359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.861404 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:54.861690 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:54.861690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:54.861691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:54.861786 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:54.861832 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:54.862155 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 03:27:54.862211 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 03:27:54.862562 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:54.852790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45373 (local address 127.6.116.130:40489)
0929 03:27:54.852970 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:54.853011 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:54.853473 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:54.853697 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:54.853700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:54.854520 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:54.854726 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:54.854731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.855500 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.855501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:54.856695 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:54.856697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:54.857096 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:54.857097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:54.857214 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:54.857735 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:54.857743 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:54.859426 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:54.860970 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:54.860976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:54.860979 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:54.861283 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:54.861478 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:54.861479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:54.861479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:54.861620 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:54.861850 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:54.861856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:54.862017 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 03:27:54.862129 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:27:54.862301 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:27:54.863119 17198 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45373
I20260929 03:27:54.863214 17198 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:54.863379 17198 heartbeater.cc:511] Master 127.6.116.190:45373 requested a full tablet report, sending...
I20260929 03:27:54.863772 16873 ts_manager.cc:194] Registered new tserver with Master: e01a8189e92f4977902aa0a093411574 (127.6.116.130:44121)
I20260929 03:27:54.864265 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:40489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:27:54.881778 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:27:54.963126 17203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:27:54.963439 17203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:27:54.963478 17203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:27:54.964705 17203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:27:54.964768 17203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:27:54.964818 17203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:27:54.966607 17203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:45373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:27:54.966869 17203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:27:54.967162 17203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:27:54.972280 17208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.972306 17211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:27:54.972321 17209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:27:54.972414 17203 server_base.cc:1034] running on GCE node
I20260929 03:27:54.972697 17203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:27:54.972935 17203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:27:54.974081 17203 hybrid_clock.cc:648] HybridClock initialized: now 1790652474974065 us; error 33 us; skew 500 ppm
Sep 29 03:27:54 dist-test-slave-z2zg krb5kdc[16828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:27:54.976575 17203 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:27:54.977028 17203 webserver.cc:460] Webserver started at http://127.6.116.131:43151/ using document root <none> and password file <none>
I20260929 03:27:54.977228 17203 fs_manager.cc:359] Metadata directory not provided
I20260929 03:27:54.977268 17203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:27:54.977355 17203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:27:54.978343 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "1413090a4e74014a3713784401a6a5b2"
server_key_iv: "9862a4608867591f303baef4a94a5aaf"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.978652 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "1413090a4e74014a3713784401a6a5b2"
server_key_iv: "9862a4608867591f303baef4a94a5aaf"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.979969 17203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:27:54.980659 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:54.980831 17203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:27:54.980937 17203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2"
format_stamp: "Formatted at 2026-09-29 03:27:54 on dist-test-slave-z2zg"
server_key: "1413090a4e74014a3713784401a6a5b2"
server_key_iv: "9862a4608867591f303baef4a94a5aaf"
server_key_version: "encryptionkey@0"
I20260929 03:27:54.981015 17203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:27:54.998113 17203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:27:55.001024 17203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:27:55.001192 17203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:27:55.001425 17203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:27:55.001747 17203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:27:55.001801 17203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:55.001843 17203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:27:55.001870 17203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:27:55.007831 17203 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:38263
I20260929 03:27:55.007869 17331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:38263 every 8 connection(s)
I20260929 03:27:55.008212 17203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:27:55 dist-test-slave-z2zg krb5kdc[16828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:27:55.016682 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 17203
I20260929 03:27:55.016805 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:27:55.017060 6610 external_mini_cluster.cc:1428] Setting key 3e392320645e2b601d39526e2b8c8f98
I20260929 03:27:55.018079 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:50059 (local address 127.6.116.190:45373)
0929 03:27:55.009029 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.009031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.009213 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:27:55.009687 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.009688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.009698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.009746 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:55.010709 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.011278 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.012413 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.012650 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.012751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.014910 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:55.014912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:55.014912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:55.014918 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:55.016317 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:55.016926 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:55.016927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:55.016928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:55.016994 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:55.017336 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:55.017337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:55.017337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:55.017482 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:55.017552 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.017934 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.018012 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 03:27:55.018473 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.008917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:45373 (local address 127.6.116.131:50059)
0929 03:27:55.009091 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.009125 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.009605 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.009806 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.009809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.010433 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:27:55.010630 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.010636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.011352 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.011352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.012315 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.012317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.012759 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.012760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.012890 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.013385 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:27:55.013394 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:27:55.014826 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:27:55.016404 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:55.016410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:55.016413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:55.016815 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:55.017072 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:27:55.017072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:27:55.017073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:27:55.017246 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:27:55.017574 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:27:55.017580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:27:55.017750 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.018176 (+ 426us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.018322 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260929 03:27:55.018904 17332 heartbeater.cc:347] Connected to a master server at 127.6.116.190:45373
I20260929 03:27:55.018986 17332 heartbeater.cc:464] Registering TS with master...
I20260929 03:27:55.019202 17332 heartbeater.cc:511] Master 127.6.116.190:45373 requested a full tablet report, sending...
I20260929 03:27:55.019603 16873 ts_manager.cc:194] Registered new tserver with Master: d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 (127.6.116.131:38263)
I20260929 03:27:55.019948 16873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:50059
I20260929 03:27:55.028836 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:27:55.035772 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.030086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.6.116.190:45373)
0929 03:27:55.030137 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.030139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.030238 (+ 99us) server_negotiation.cc:407] Connection header received
0929 03:27:55.030296 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.030296 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.030304 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.030350 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:27:55.030872 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.031400 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.032075 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.032243 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.032289 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.033375 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:27:55.033376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:27:55.033376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:27:55.033381 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:27:55.034863 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:55.035208 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:55.035209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:55.035209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:55.035260 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:27:55.035381 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:27:55.035382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:27:55.035382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:27:55.035504 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:27:55.035585 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.035658 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.035714 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:27:55.037173 16873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:27:55.038895 16873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:27:55.045285 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:44121 (local address 127.0.0.1:58804)
0929 03:27:55.041550 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.041562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.041619 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.041865 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.041866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.041868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:55.041918 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.041919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.043288 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.043289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.044313 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.044315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.045049 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.045050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.045101 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.045104 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.045150 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.045188 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260929 03:27:55.045300 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.6.116.129:40777)
0929 03:27:55.041569 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.041572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.041581 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:27:55.041623 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.041624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.041636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.041727 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:55.041913 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.042660 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044132 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.044867 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044940 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.044998 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.045162 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.045213 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":86,"threads_started":1}
I20260929 03:27:55.045514 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38263 (local address 127.0.0.1:43660)
0929 03:27:55.041697 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.041705 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.041747 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.042061 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.042061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.042063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:55.042093 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.042094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.043323 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.043324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.044357 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.044359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.045292 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.045293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.045357 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.045362 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.045414 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.045455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":62,"threads_started":1}
I20260929 03:27:55.045614 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.6.116.130:44121)
0929 03:27:55.041534 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.041538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.041605 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:27:55.041687 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.041689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.041708 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.041784 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:55.042012 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.043187 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044401 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.044877 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044962 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.045018 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.045487 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.045531 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":112,"threads_started":1}
I20260929 03:27:55.045758 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.6.116.131:38263)
0929 03:27:55.041517 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.041523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.041854 (+ 331us) server_negotiation.cc:407] Connection header received
0929 03:27:55.041902 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.041905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.041919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.041972 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:55.042161 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.043244 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044448 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.045149 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.045212 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.045259 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.045626 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.045681 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260929 03:27:55.045284 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.041085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:40777 (local address 127.0.0.1:33884)
0929 03:27:55.041256 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.041296 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.041359 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.041732 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.041736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.041738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:55.041801 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.041803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.042758 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.042759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.044053 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.044056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.044964 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.044964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.045039 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.045052 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.045132 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.045188 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":45,"threads_started":1}
I20260929 03:27:55.046330 17132 tablet_service.cc:1467] Processing CreateTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 (DEFAULT_TABLE table=test-table [id=258bb1e41e754f55b47a1c6da85285bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:55.046608 17132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b799cf7bfb4976ae3e678d8b8e5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:55.046648 17266 tablet_service.cc:1467] Processing CreateTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 (DEFAULT_TABLE table=test-table [id=258bb1e41e754f55b47a1c6da85285bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:55.046790 16998 tablet_service.cc:1467] Processing CreateTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 (DEFAULT_TABLE table=test-table [id=258bb1e41e754f55b47a1c6da85285bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:27:55.046955 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b799cf7bfb4976ae3e678d8b8e5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:55.047091 16998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b799cf7bfb4976ae3e678d8b8e5be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:27:55.049072 17349 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Bootstrap starting.
I20260929 03:27:55.049623 17349 tablet_bootstrap.cc:654] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:55.050161 17351 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Bootstrap starting.
I20260929 03:27:55.050195 17349 log.cc:824] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:55.050819 17351 tablet_bootstrap.cc:654] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:55.050978 17349 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: No bootstrap required, opened a new log
I20260929 03:27:55.051097 17349 ts_tablet_manager.cc:1397] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:27:55.051393 17351 log.cc:824] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:55.051718 17350 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Bootstrap starting.
I20260929 03:27:55.052093 17351 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: No bootstrap required, opened a new log
I20260929 03:27:55.052178 17351 ts_tablet_manager.cc:1397] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:27:55.052317 17350 tablet_bootstrap.cc:654] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Neither blocks nor log segments found. Creating new log.
I20260929 03:27:55.052862 17350 log.cc:824] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:27:55.053294 17349 raft_consensus.cc:348] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.053416 17349 raft_consensus.cc:374] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:55.053435 17349 raft_consensus.cc:729] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01a8189e92f4977902aa0a093411574, State: Initialized, Role: FOLLOWER
I20260929 03:27:55.053551 17349 consensus_queue.cc:260] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.053828 17349 ts_tablet_manager.cc:1428] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:27:55.053865 17198 heartbeater.cc:503] Master 127.6.116.190:45373 was elected leader, sending a full tablet report...
I20260929 03:27:55.053982 17351 raft_consensus.cc:348] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.054152 17351 raft_consensus.cc:374] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:55.054194 17351 raft_consensus.cc:729] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a7e0e1ba3d4e8988e1aa3e53a9a4a2, State: Initialized, Role: FOLLOWER
I20260929 03:27:55.054311 17351 consensus_queue.cc:260] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.054436 17350 tablet_bootstrap.cc:492] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: No bootstrap required, opened a new log
I20260929 03:27:55.054490 17350 ts_tablet_manager.cc:1397] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:27:55.054597 17351 ts_tablet_manager.cc:1428] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:27:55.054723 17332 heartbeater.cc:503] Master 127.6.116.190:45373 was elected leader, sending a full tablet report...
I20260929 03:27:55.056540 17350 raft_consensus.cc:348] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.056696 17350 raft_consensus.cc:374] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:27:55.056725 17350 raft_consensus.cc:729] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e90d8dca90d4834988cf150840991f0, State: Initialized, Role: FOLLOWER
I20260929 03:27:55.056797 17350 consensus_queue.cc:260] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.057031 17350 ts_tablet_manager.cc:1428] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:27:55.057096 17064 heartbeater.cc:503] Master 127.6.116.190:45373 was elected leader, sending a full tablet report...
W20260929 03:27:55.102665 17199 tablet.cc:2369] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:27:55.204116 17065 tablet.cc:2369] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:55.228542 17357 raft_consensus.cc:482] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:27:55.228718 17357 raft_consensus.cc:504] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.229080 17357 leader_election.cc:288] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 (127.6.116.131:38263), e01a8189e92f4977902aa0a093411574 (127.6.116.130:44121)
I20260929 03:27:55.232064 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.229309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:59553 (local address 127.6.116.130:44121)
0929 03:27:55.229358 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.229360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.229493 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:27:55.229556 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.229557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.229568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.229621 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:55.229778 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.230346 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231328 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.231697 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231798 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.231805 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.231937 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.231988 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:27:55.232079 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.229281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43561 (local address 127.6.116.131:38263)
0929 03:27:55.229344 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.229346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.229444 (+ 98us) server_negotiation.cc:407] Connection header received
0929 03:27:55.229515 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.229516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.229526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.229597 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:27:55.229746 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.230277 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231377 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.231724 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.231822 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.231980 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.232011 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:27:55.232101 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.229229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38263 (local address 127.6.116.129:43561)
0929 03:27:55.229369 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.229378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.229442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.229601 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.229602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.229604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:55.229677 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.229678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.230364 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.230365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.231280 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.231282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.231815 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.231885 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.231888 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.231973 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.232011 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260929 03:27:55.232085 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.229260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:44121 (local address 127.6.116.129:59553)
0929 03:27:55.229419 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.229427 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:27:55.229501 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:27:55.229625 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:27:55.229625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:27:55.229627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:27:55.229711 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.229713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.230438 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.230439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.231239 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:27:55.231241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.231798 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.231799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:27:55.231863 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.231868 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:27:55.231933 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:27:55.231986 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260929 03:27:55.232404 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b799cf7bfb4976ae3e678d8b8e5be7" candidate_uuid: "7e90d8dca90d4834988cf150840991f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" is_pre_election: true
I20260929 03:27:55.232586 17286 raft_consensus.cc:2454] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e90d8dca90d4834988cf150840991f0 in term 0.
I20260929 03:27:55.232821 16953 leader_election.cc:302] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e90d8dca90d4834988cf150840991f0, d7a7e0e1ba3d4e8988e1aa3e53a9a4a2; no voters:
I20260929 03:27:55.232810 17152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b799cf7bfb4976ae3e678d8b8e5be7" candidate_uuid: "7e90d8dca90d4834988cf150840991f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01a8189e92f4977902aa0a093411574" is_pre_election: true
I20260929 03:27:55.232957 17357 raft_consensus.cc:2781] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:27:55.232995 17152 raft_consensus.cc:2454] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e90d8dca90d4834988cf150840991f0 in term 0.
I20260929 03:27:55.233016 17357 raft_consensus.cc:482] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:27:55.233047 17357 raft_consensus.cc:3037] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:55.233811 17357 raft_consensus.cc:504] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.233945 17357 leader_election.cc:288] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [CANDIDATE]: Term 1 election: Requested vote from peers d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 (127.6.116.131:38263), e01a8189e92f4977902aa0a093411574 (127.6.116.130:44121)
I20260929 03:27:55.234086 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b799cf7bfb4976ae3e678d8b8e5be7" candidate_uuid: "7e90d8dca90d4834988cf150840991f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2"
I20260929 03:27:55.234153 17286 raft_consensus.cc:3037] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:55.234141 17152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b799cf7bfb4976ae3e678d8b8e5be7" candidate_uuid: "7e90d8dca90d4834988cf150840991f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e01a8189e92f4977902aa0a093411574"
I20260929 03:27:55.234220 17152 raft_consensus.cc:3037] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:27:55.234819 17286 raft_consensus.cc:2454] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e90d8dca90d4834988cf150840991f0 in term 1.
I20260929 03:27:55.234997 16953 leader_election.cc:302] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e90d8dca90d4834988cf150840991f0, d7a7e0e1ba3d4e8988e1aa3e53a9a4a2; no voters:
I20260929 03:27:55.235069 17152 raft_consensus.cc:2454] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e90d8dca90d4834988cf150840991f0 in term 1.
I20260929 03:27:55.235109 17357 raft_consensus.cc:2781] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:27:55.235272 17357 raft_consensus.cc:686] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7e90d8dca90d4834988cf150840991f0, State: Running, Role: LEADER
I20260929 03:27:55.235388 17357 consensus_queue.cc:237] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } }
I20260929 03:27:55.236155 16872 catalog_manager.cc:5495] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e90d8dca90d4834988cf150840991f0 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "7e90d8dca90d4834988cf150840991f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e90d8dca90d4834988cf150840991f0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 } health_report { overall_health: UNKNOWN } } }
W20260929 03:27:55.258808 17333 tablet.cc:2369] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:27:55.275199 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.272648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.6.116.129:40777)
0929 03:27:55.272691 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:27:55.272711 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 03:27:55.272721 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:27:55.272785 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:27:55.272786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:27:55.272807 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:27:55.272896 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:27:55.273029 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.273534 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.274333 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:27:55.274568 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:27:55.274658 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:27:55.274830 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:27:55.274941 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:27:55.275035 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:27:55.275092 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:27:55.275112 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:27:55.277104 17152 raft_consensus.cc:1261] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 1 FOLLOWER]: Refusing update from remote peer 7e90d8dca90d4834988cf150840991f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:55.277103 17286 raft_consensus.cc:1261] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 1 FOLLOWER]: Refusing update from remote peer 7e90d8dca90d4834988cf150840991f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:27:55.277467 17357 consensus_queue.cc:1035] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e01a8189e92f4977902aa0a093411574" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 44121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:55.277629 17360 consensus_queue.cc:1035] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7a7e0e1ba3d4e8988e1aa3e53a9a4a2" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:27:55.280306 17364 mvcc.cc:205] Tried to move back new op lower bound from 7334512538732220416 to 7334512538564702208. Current Snapshot: MvccSnapshot[applied={T|T < 7334512538732220416}]
I20260929 03:27:55.280498 17363 mvcc.cc:205] Tried to move back new op lower bound from 7334512538732220416 to 7334512538564702208. Current Snapshot: MvccSnapshot[applied={T|T < 7334512538732220416 or (T in {7334512538732220416})}]
I20260929 03:27:55.282048 17365 mvcc.cc:205] Tried to move back new op lower bound from 7334512538732220416 to 7334512538564702208. Current Snapshot: MvccSnapshot[applied={T|T < 7334512538732220416 or (T in {7334512538732220416})}]
I20260929 03:27:55.282236 16873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:55.282328 16873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:27:55.283269 16873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e13bb9b564d5431b950cfd67964c8c6b: Sending DeleteTablet for 3 replicas of tablet f9b799cf7bfb4976ae3e678d8b8e5be7
I20260929 03:27:55.283596 16998 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:55 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:33884
I20260929 03:27:55.283654 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:55 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:43660
I20260929 03:27:55.283664 17132 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b799cf7bfb4976ae3e678d8b8e5be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:27:55 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:58804
I20260929 03:27:55.283857 17372 tablet_replica.cc:330] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0: stopping tablet replica
I20260929 03:27:55.283967 17372 raft_consensus.cc:2229] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:27:55.284055 17372 raft_consensus.cc:2258] T f9b799cf7bfb4976ae3e678d8b8e5be7 P 7e90d8dca90d4834988cf150840991f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:55.284111 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16935
I20260929 03:27:55.284616 17374 tablet_replica.cc:330] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: stopping tablet replica
I20260929 03:27:55.284735 17374 raft_consensus.cc:2229] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:55.284838 17374 raft_consensus.cc:2258] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:55.285220 17374 ts_tablet_manager.cc:1905] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:55.286736 17374 ts_tablet_manager.cc:1918] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:55.286793 17374 log.cc:1196] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/f9b799cf7bfb4976ae3e678d8b8e5be7
I20260929 03:27:55.287099 17374 ts_tablet_manager.cc:1939] T f9b799cf7bfb4976ae3e678d8b8e5be7 P e01a8189e92f4977902aa0a093411574: Deleting consensus metadata
I20260929 03:27:55.287508 16860 catalog_manager.cc:4841] TS e01a8189e92f4977902aa0a093411574 (127.6.116.130:44121): tablet f9b799cf7bfb4976ae3e678d8b8e5be7 (table test-table [id=258bb1e41e754f55b47a1c6da85285bc]) successfully deleted
W20260929 03:27:55.290654 16859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:40777 (error 108)
I20260929 03:27:55.290876 17373 tablet_replica.cc:330] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: stopping tablet replica
I20260929 03:27:55.291008 17373 raft_consensus.cc:2229] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:27:55.291143 17373 raft_consensus.cc:2258] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:27:55.291210 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:27:55.291096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:40777 (local address 127.0.0.1:33894)
0929 03:27:55.291144 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:27:55.291178 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:40777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:27:55.291325 16859 catalog_manager.cc:4584] TS 7e90d8dca90d4834988cf150840991f0 (127.6.116.129:40777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9b799cf7bfb4976ae3e678d8b8e5be7: Network error: Client connection negotiation failed: client connection to 127.6.116.129:40777: connect: Connection refused (error 111)
I20260929 03:27:55.291359 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 17069
I20260929 03:27:55.292116 17373 ts_tablet_manager.cc:1905] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:27:55.294075 17373 ts_tablet_manager.cc:1918] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:27:55.294155 17373 log.cc:1196] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/f9b799cf7bfb4976ae3e678d8b8e5be7
I20260929 03:27:55.294427 17373 ts_tablet_manager.cc:1939] T f9b799cf7bfb4976ae3e678d8b8e5be7 P d7a7e0e1ba3d4e8988e1aa3e53a9a4a2: Deleting consensus metadata
I20260929 03:27:55.294893 16859 catalog_manager.cc:4841] TS d7a7e0e1ba3d4e8988e1aa3e53a9a4a2 (127.6.116.131:38263): tablet f9b799cf7bfb4976ae3e678d8b8e5be7 (table test-table [id=258bb1e41e754f55b47a1c6da85285bc]) successfully deleted
I20260929 03:27:55.298066 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 17203
I20260929 03:27:55.302461 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 16841
2026-09-29T03:27:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2061 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:27:55 dist-test-slave-z2zg krb5kdc[17378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:27:55 dist-test-slave-z2zg krb5kdc[17378](info): set up 2 sockets
Sep 29 03:27:55 dist-test-slave-z2zg krb5kdc[17378](info): commencing operation
krb5kdc: starting...
W20260929 03:27:56.321898 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:27:56 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:27:56Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangeradmin_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangeradmin_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangerlookup_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangerlookup_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
I20260929 03:27:56.479597 6610 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFuIZmB/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/postgres -l logfile start
2026-09-29 03:27:58.858 UTC [17408] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:27:58.858 UTC [17408] LOG: listening on IPv4 address "127.6.116.148", port 39013
2026-09-29 03:27:58.861 UTC [17408] LOG: listening on Unix socket "/tmp/.s.PGSQL.39013"
2026-09-29 03:27:58.881 UTC [17411] LOG: database system was shut down at 2026-09-29 03:27:57 UTC
2026-09-29 03:27:58.884 UTC [17408] LOG: database system is ready to accept connections
I20260929 03:27:59.845947 6610 mini_postgres.cc:96] Postgres bound to 39013
2026-09-29 03:27:59.851 UTC [17419] FATAL: database "slave" does not exist
127.6.116.148:39013 - accepting connections
I20260929 03:27:59.852492 6610 mini_ranger.cc:161] Starting Ranger...
I20260929 03:27:59.867270 6610 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:28:00.316812 6610 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:28:00.316881 6610 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin
2026-09-29 03:28:00,590 [I] DB FLAVOR :POSTGRES
2026-09-29 03:28:00,590 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:28:00,590 [I] Checking connection..
2026-09-29 03:28:00,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:28:00,843 [I] Checking connection passed.
2026-09-29 03:28:00,844 [I] --------- Verifying version history table ---------
2026-09-29 03:28:00,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:01,097 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:28:01,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:01,351 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:28:01,351 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:28:01,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:28:01.567 UTC [17483] WARNING: there is no transaction in progress
2026-09-29 03:28:01,573 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:28:01,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:01,818 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:28:01,819 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:28:01,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:28:02,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:28:02,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z2zg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:28:02,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:28:02,762 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:28:02,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:28:03,021 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:28:03,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:28:03,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:28:03,465 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:28:03,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:28:04.360 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.375 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.390 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.404 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.421 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.440 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.491 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.497 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.510 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.519 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.528 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.540 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.550 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.556 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.562 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:04.568 UTC [17591] WARNING: there is no transaction in progress
2026-09-29 03:28:05,156 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:28:05,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:28:05,399 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:28:05,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:28:05,641 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:28:05,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:28:05,867 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:28:05,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:28:06,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:28:06,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z2zg.c.gcp-upstream.internal';"
2026-09-29 03:28:06,569 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:28:06,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:28:06,822 [I] DB_PATCHES have already been applied
I20260929 03:28:06.828143 6610 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFuIZmB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:28:06.828203 6610 mini_ranger.cc:193] Using host: 127.6.116.148
I20260929 03:28:06.831149 6610 mini_ranger.cc:227] Ranger admin URL: http://127.6.116.148:45361
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:28:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:28:07.933279 6610 mini_ranger.cc:160] Time spent starting Ranger: real 8.081s user 0.001s sys 0.007s
Sep 29, 2026 3:28:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:28:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 29, 2026 3:28:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:28:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:28:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:28:29.387261 6610 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangerkms_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/rangerkms_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:28:29 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:28:29.469187 6610 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:28:29.477901 6610 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:28:30.251931 6610 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:28:30.252017 6610 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms
I20260929 03:28:30.252034 6610 mini_ranger_kms.cc:227] Using postgres at 127.6.116.148:39013
2026-09-29 03:28:30,510 [I] DB FLAVOR :POSTGRES
2026-09-29 03:28:30,511 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:28:30,511 [I] Checking connection
2026-09-29 03:28:30,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:28:30,773 [I] connection success
2026-09-29 03:28:30,774 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:28:30,774 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:28:30,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:28:31,010 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:28:31,011 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:28:31,011 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:28:31,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:39013/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:28:31,292 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:28:31.743249 6610 mini_ranger_kms.cc:310] Created kms service
I20260929 03:28:31.925567 6610 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:28:31.966356 6610 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:28:31 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652509, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:32.334403 6610 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:28:32.334512 6610 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:28:32.334539 6610 mini_ranger_kms.cc:240] Using host: 127.6.116.148
I20260929 03:28:32.338272 6610 mini_ranger_kms.cc:276] Ranger KMS PID: 17771
I20260929 03:28:32.338392 6610 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.116.148:51245
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:28:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:28:34.509974 6610 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.041s user 0.000s sys 0.241s
I20260929 03:28:34.510084 6610 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:28:34.510171 6610 mini_ranger_kms.cc:401] 127.6.116.148:51245/kms/v1/keys
Sep 29 03:28:34 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:28:34 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
Sep 29 03:28:35 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652514, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
Sep 29 03:28:37 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:28:37.440660 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39189
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:39189
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:51245/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:28:37.539603 17822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:28:37.539786 17822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:28:37.539806 17822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:28:37.540921 17822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:28:37.540968 17822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:28:37.540983 17822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:28:37.540997 17822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:28:37.541040 17822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:28:37.542472 17822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.116.148:51245/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:39189
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:28:37.542716 17822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:28:37.542973 17822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:28:37.548552 17830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:37.548617 17828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:37.548561 17827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:28:37.548902 17822 server_base.cc:1034] running on GCE node
I20260929 03:28:37.549146 17822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:28:37.549429 17822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:28:37.550580 17822 hybrid_clock.cc:648] HybridClock initialized: now 1790652517550570 us; error 26 us; skew 500 ppm
Sep 29 03:28:37 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:28:37.558149 17822 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:28:37.558630 17822 webserver.cc:460] Webserver started at http://127.6.116.190:40519/ using document root <none> and password file <none>
I20260929 03:28:37.558872 17822 fs_manager.cc:359] Metadata directory not provided
I20260929 03:28:37.558931 17822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:28:37.559075 17822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:28:37 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652517, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:37.605770 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "3725bb7e01244f8b966da3227b019b5a"
format_stamp: "Formatted at 2026-09-29 03:28:37 on dist-test-slave-z2zg"
server_key: "3c36cdc689037f3c19c4c8792568a113"
server_key_iv: "8fc38a2ef58b990570c61e46f52e8a29"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:37.606173 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "3725bb7e01244f8b966da3227b019b5a"
format_stamp: "Formatted at 2026-09-29 03:28:37 on dist-test-slave-z2zg"
server_key: "3c36cdc689037f3c19c4c8792568a113"
server_key_iv: "8fc38a2ef58b990570c61e46f52e8a29"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:37.607693 17822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:28:37.621465 17837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:37.621767 17822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:28:37.621937 17822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "3725bb7e01244f8b966da3227b019b5a"
format_stamp: "Formatted at 2026-09-29 03:28:37 on dist-test-slave-z2zg"
server_key: "3c36cdc689037f3c19c4c8792568a113"
server_key_iv: "8fc38a2ef58b990570c61e46f52e8a29"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:37.622035 17822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:28:37.634609 17822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:28:37.636516 17822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:28:37.636657 17822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:28:37.641709 17822 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:39189
I20260929 03:28:37.641734 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:39189 every 8 connection(s)
I20260929 03:28:37.642128 17822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:28:37.642781 17890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:28:37.644996 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: Bootstrap starting.
I20260929 03:28:37.645923 17890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: Neither blocks nor log segments found. Creating new log.
I20260929 03:28:37.646093 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 17822
I20260929 03:28:37.646198 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:28:37.646263 17890 log.cc:824] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:28:37.646369 6610 external_mini_cluster.cc:1428] Setting key 161ce7eca329551633eee2530f428b39
I20260929 03:28:37.646986 17890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: No bootstrap required, opened a new log
I20260929 03:28:37.649015 17890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } }
I20260929 03:28:37.649168 17890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:28:37.649241 17890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3725bb7e01244f8b966da3227b019b5a, State: Initialized, Role: FOLLOWER
I20260929 03:28:37.649364 17890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } }
I20260929 03:28:37.649471 17890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:28:37.649524 17890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:28:37.649573 17890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:28:37.650049 17890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } }
I20260929 03:28:37.650161 17890 leader_election.cc:302] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3725bb7e01244f8b966da3227b019b5a; no voters:
I20260929 03:28:37.650305 17890 leader_election.cc:288] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:28:37.650568 17890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:28:37.651499 17896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:28:37.651706 17896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [term 1 LEADER]: Becoming Leader. State: Replica: 3725bb7e01244f8b966da3227b019b5a, State: Running, Role: LEADER
I20260929 03:28:37.651839 17896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } }
I20260929 03:28:37.652462 17896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3725bb7e01244f8b966da3227b019b5a. Latest consensus state: current_term: 1 leader_uuid: "3725bb7e01244f8b966da3227b019b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } } }
I20260929 03:28:37.652601 17896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [sys.catalog]: This master's current role is: LEADER
I20260929 03:28:37.652722 17890 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:28:37.652843 17909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:28:37.653319 17909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:28:37.653285 17895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3725bb7e01244f8b966da3227b019b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3725bb7e01244f8b966da3227b019b5a" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39189 } } }
I20260929 03:28:37.653463 17895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a [sys.catalog]: This master's current role is: LEADER
Sep 29 03:28:37 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:28:37.661857 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:37.646823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.6.116.190:39189)
0929 03:28:37.646982 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:37.646991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:37.647020 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:28:37.647540 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:37.647548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:37.647608 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:37.647751 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:28:37.648494 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:37.650244 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:37.651403 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:37.652032 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:37.652142 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:37.656185 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:28:37.656241 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:28:37.656242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:28:37.656348 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:28:37.659947 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:37.660347 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:37.660350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:37.660351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:37.660463 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:37.660694 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:37.660695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:37.660696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:37.661076 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:28:37.661256 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:37.661354 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 03:28:37.661592 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260929 03:28:37.664366 17909 catalog_manager.cc:1347] Generated new cluster ID: fff61e3cd26046edbf4ab294d33e6b29
I20260929 03:28:37.664464 17909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:28:37.675025 17909 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:28:37.675741 17909 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:28:37.687000 17909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: Generated new TSK 0
I20260929 03:28:37.687237 17909 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:28:38.419500 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:51245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:28:38.527401 17935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:28:38.527637 17935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:28:38.527668 17935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:28:38.529217 17935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:28:38.529275 17935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:28:38.529313 17935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:28:38.531348 17935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.116.148:51245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:28:38.531569 17935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:28:38.531853 17935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:28:38.538115 17943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:38.538120 17941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:38.538125 17940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:28:38.538406 17935 server_base.cc:1034] running on GCE node
I20260929 03:28:38.538609 17935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:28:38.538874 17935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:28:38.540046 17935 hybrid_clock.cc:648] HybridClock initialized: now 1790652518540029 us; error 35 us; skew 500 ppm
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:28:38.543380 17935 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:28:38.543895 17935 webserver.cc:460] Webserver started at http://127.6.116.129:41825/ using document root <none> and password file <none>
I20260929 03:28:38.544144 17935 fs_manager.cc:359] Metadata directory not provided
I20260929 03:28:38.544200 17935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:28:38.544313 17935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:38.556947 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "017c23186a8a42d2a295d16bfd49374c"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "14445185e32f449515fa4533d66e1900"
server_key_iv: "569f81ce35fa2c85d1753d3c5838a154"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.557369 17935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "017c23186a8a42d2a295d16bfd49374c"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "14445185e32f449515fa4533d66e1900"
server_key_iv: "569f81ce35fa2c85d1753d3c5838a154"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.559362 17935 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:28:38.570511 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.570731 17935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:28:38.570825 17935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "017c23186a8a42d2a295d16bfd49374c"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "14445185e32f449515fa4533d66e1900"
server_key_iv: "569f81ce35fa2c85d1753d3c5838a154"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.570909 17935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:28:38.583303 17935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:28:38.585623 17935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:28:38.585810 17935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:28:38.586102 17935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:28:38.586504 17935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:28:38.586611 17935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.586707 17935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:28:38.586786 17935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.593986 17935 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:43681
I20260929 03:28:38.594131 18065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:43681 every 8 connection(s)
I20260929 03:28:38.594508 17935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:28:38.595841 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 17935
I20260929 03:28:38.595955 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:28:38.596172 6610 external_mini_cluster.cc:1428] Setting key 3e6e7bafc9056ebf3fd06f19fc44332a
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:28:38.606158 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:38.595222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39189 (local address 127.6.116.129:41725)
0929 03:28:38.595416 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:38.595473 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:38.596084 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:38.596470 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:38.596474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:38.596694 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:28:38.596951 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:38.596957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.597933 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.597934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:38.599204 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:38.599207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.599763 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.599764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:38.599900 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:38.600399 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:28:38.600427 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:28:38.602403 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:28:38.604527 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:38.604533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:38.604535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:38.604825 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:38.605057 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:38.605058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:38.605058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:38.605186 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:38.605494 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:28:38.605500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:28:38.605650 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 03:28:38.605759 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:28:38.605910 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260929 03:28:38.606693 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:38.595263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:41725 (local address 127.6.116.190:39189)
0929 03:28:38.595503 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:38.595506 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:38.595552 (+ 46us) server_negotiation.cc:407] Connection header received
0929 03:28:38.596268 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:38.596269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:38.596282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:38.596344 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:28:38.597055 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.597832 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.599308 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.599620 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.599700 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:38.602515 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:28:38.602517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:28:38.602517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:28:38.602524 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:28:38.604415 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:38.604915 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:38.604916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:38.604917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:38.604985 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:38.605258 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:38.605259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:38.605259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:38.605394 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:28:38.605467 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:38.606510 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0929 03:28:38.606596 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260929 03:28:38.607427 18066 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39189
I20260929 03:28:38.607529 18066 heartbeater.cc:464] Registering TS with master...
I20260929 03:28:38.607697 18066 heartbeater.cc:511] Master 127.6.116.190:39189 requested a full tablet report, sending...
I20260929 03:28:38.608250 17854 ts_manager.cc:194] Registered new tserver with Master: 017c23186a8a42d2a295d16bfd49374c (127.6.116.129:43681)
I20260929 03:28:38.609115 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:41725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:28:38.632215 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:51245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:28:38.747784 18072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:28:38.748134 18072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:28:38.748220 18072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:28:38.749895 18072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:28:38.749995 18072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:28:38.750073 18072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:28:38.752178 18072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.116.148:51245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:28:38.752769 18072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:28:38.753068 18072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:28:38.759330 18080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:38.759632 18079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:38.759874 18082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:28:38.760596 18072 server_base.cc:1034] running on GCE node
I20260929 03:28:38.760905 18072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:28:38.761160 18072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:28:38.762331 18072 hybrid_clock.cc:648] HybridClock initialized: now 1790652518762310 us; error 34 us; skew 500 ppm
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:28:38.765473 18072 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:28:38.765920 18072 webserver.cc:460] Webserver started at http://127.6.116.130:45155/ using document root <none> and password file <none>
I20260929 03:28:38.766158 18072 fs_manager.cc:359] Metadata directory not provided
I20260929 03:28:38.766206 18072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:28:38.766319 18072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:38.780498 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "115b801c7a194581aae6551930310b38"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "37652f35055fb373346a6edb56e1aa04"
server_key_iv: "d7812f810e854f9ee92ac9bc528ea5f8"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.781005 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "115b801c7a194581aae6551930310b38"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "37652f35055fb373346a6edb56e1aa04"
server_key_iv: "d7812f810e854f9ee92ac9bc528ea5f8"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.784400 18072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 03:28:38.798700 18089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.798918 18072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260929 03:28:38.798997 18072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "115b801c7a194581aae6551930310b38"
format_stamp: "Formatted at 2026-09-29 03:28:38 on dist-test-slave-z2zg"
server_key: "37652f35055fb373346a6edb56e1aa04"
server_key_iv: "d7812f810e854f9ee92ac9bc528ea5f8"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:38.799088 18072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:28:38.837153 18072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:28:38.839448 18072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:28:38.839632 18072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:28:38.839877 18072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:28:38.840235 18072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:28:38.840322 18072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.840498 18072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:28:38.840605 18072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:38.850317 18072 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:38751
I20260929 03:28:38.851106 18202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:38751 every 8 connection(s)
I20260929 03:28:38.851013 18072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:28:38.857693 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 18072
I20260929 03:28:38.857789 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:28:38.857988 6610 external_mini_cluster.cc:1428] Setting key 1d4f051f2f7599591e4044f17ccb802e
Sep 29 03:28:38 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:28:38.861874 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:38.852288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:49101 (local address 127.6.116.190:39189)
0929 03:28:38.852361 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:38.852363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:38.852579 (+ 216us) server_negotiation.cc:407] Connection header received
0929 03:28:38.853129 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:38.853130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:38.853142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:38.853222 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:28:38.853904 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.854603 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.855994 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.856312 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.856385 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:38.858687 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:28:38.858688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:28:38.858688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:28:38.858696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:28:38.860401 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:38.860860 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:38.860861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:38.860861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:38.860923 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:38.861189 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:38.861190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:38.861190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:38.861326 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:28:38.861391 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:38.861688 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 03:28:38.861783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:28:38.862023 18205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:38.852271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39189 (local address 127.6.116.130:49101)
0929 03:28:38.852437 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:38.852482 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:38.853032 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:38.853291 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:38.853295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:38.853530 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:28:38.853792 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:38.853798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.854714 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.854715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:38.855904 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:38.855906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:38.856407 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:38.856408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:38.856543 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:38.857046 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:28:38.857056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:28:38.858580 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:28:38.860473 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:38.860479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:38.860481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:38.860780 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:38.860996 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:38.860997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:38.860997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:38.861121 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:38.861393 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:28:38.861398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:28:38.861547 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:28:38.861658 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:28:38.861816 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260929 03:28:38.862579 18203 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39189
I20260929 03:28:38.862673 18203 heartbeater.cc:464] Registering TS with master...
I20260929 03:28:38.862866 18203 heartbeater.cc:511] Master 127.6.116.190:39189 requested a full tablet report, sending...
I20260929 03:28:38.863267 17854 ts_manager.cc:194] Registered new tserver with Master: 115b801c7a194581aae6551930310b38 (127.6.116.130:38751)
I20260929 03:28:38.863725 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:49101
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:28:38.892277 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:51245/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:28:39.014170 18209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:28:39.014490 18209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:28:39.014570 18209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:28:39.016168 18209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:28:39.016260 18209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:28:39.016345 18209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:28:39.018366 18209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.116.148:51245/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:28:39.018779 18209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:28:39.019086 18209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:28:39.025395 18214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:39.025395 18215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:28:39.025722 18217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:28:39.026603 18209 server_base.cc:1034] running on GCE node
I20260929 03:28:39.026892 18209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:28:39.027174 18209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:28:39.028332 18209 hybrid_clock.cc:648] HybridClock initialized: now 1790652519028315 us; error 31 us; skew 500 ppm
Sep 29 03:28:39 dist-test-slave-z2zg krb5kdc[17378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:28:39.031610 18209 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:28:39.032178 18209 webserver.cc:460] Webserver started at http://127.6.116.131:43081/ using document root <none> and password file <none>
I20260929 03:28:39.032437 18209 fs_manager.cc:359] Metadata directory not provided
I20260929 03:28:39.032485 18209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:28:39.032598 18209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:28:39 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:39.049541 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "d759088772794b8a90ebf62b1c8eb4b8"
format_stamp: "Formatted at 2026-09-29 03:28:39 on dist-test-slave-z2zg"
server_key: "b4f5a2da9fe0f6f28f0b5e34830f024d"
server_key_iv: "1046e8d4abe8f6c2148fb73e489ebd2a"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:39.050024 18209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "d759088772794b8a90ebf62b1c8eb4b8"
format_stamp: "Formatted at 2026-09-29 03:28:39 on dist-test-slave-z2zg"
server_key: "b4f5a2da9fe0f6f28f0b5e34830f024d"
server_key_iv: "1046e8d4abe8f6c2148fb73e489ebd2a"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:39.058482 18209 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.001s
I20260929 03:28:39.072608 18224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:39.072798 18209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 03:28:39.072871 18209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "d759088772794b8a90ebf62b1c8eb4b8"
format_stamp: "Formatted at 2026-09-29 03:28:39 on dist-test-slave-z2zg"
server_key: "b4f5a2da9fe0f6f28f0b5e34830f024d"
server_key_iv: "1046e8d4abe8f6c2148fb73e489ebd2a"
server_key_version: "kuduclusterkey@0"
I20260929 03:28:39.072942 18209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:28:39.089196 18209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:28:39.091634 18209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:28:39.091835 18209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:28:39.092101 18209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:28:39.092517 18209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:28:39.092610 18209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:39.092680 18209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:28:39.092737 18209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:28:39.101212 18209 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:35751
I20260929 03:28:39.101653 18209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:28:39.102985 18337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:35751 every 8 connection(s)
I20260929 03:28:39.110021 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 18209
I20260929 03:28:39.110131 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
Sep 29 03:28:39 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652519, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:28:39.110316 6610 external_mini_cluster.cc:1428] Setting key 9edf88f0b5cadcd8a521741ea9252867
I20260929 03:28:39.114279 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.104518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:43997 (local address 127.6.116.190:39189)
0929 03:28:39.104597 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.104599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.104608 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:28:39.104633 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.104633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.104643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.104705 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:28:39.105567 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.106339 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.107825 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.108086 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.108167 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.110810 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:28:39.110812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:28:39.110812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:28:39.110821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:28:39.112744 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:39.113209 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:39.113210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:39.113211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:39.113280 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:39.113561 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:39.113562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:39.113562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:39.113700 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:28:39.113768 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.114102 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.114189 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 03:28:39.114566 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.102717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39189 (local address 127.6.116.131:43997)
0929 03:28:39.103240 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.103278 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.103861 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.104807 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.104812 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.105018 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:28:39.105434 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.105441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.106453 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.106455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.107725 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.107727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.108183 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.108184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.108321 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.108834 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:28:39.108844 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:28:39.110652 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:28:39.112841 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:39.112848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:39.112850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:39.113128 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:39.113357 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:28:39.113358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:28:39.113358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:28:39.113490 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:28:39.113773 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:28:39.113779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:28:39.113939 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.114051 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.114196 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":74,"threads_started":1}
I20260929 03:28:39.114928 18338 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39189
I20260929 03:28:39.115011 18338 heartbeater.cc:464] Registering TS with master...
I20260929 03:28:39.115250 18338 heartbeater.cc:511] Master 127.6.116.190:39189 requested a full tablet report, sending...
I20260929 03:28:39.115581 17854 ts_manager.cc:194] Registered new tserver with Master: d759088772794b8a90ebf62b1c8eb4b8 (127.6.116.131:35751)
I20260929 03:28:39.115990 17854 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:43997
I20260929 03:28:39.122283 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:28:39.130865 18069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.123981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.6.116.190:39189)
0929 03:28:39.124042 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.124044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.124055 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:28:39.124078 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.124079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.124090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.124190 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:28:39.124665 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.125401 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.126247 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.126510 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.126583 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.127869 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:28:39.127870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:28:39.127871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:28:39.127883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:28:39.129684 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:39.130058 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:39.130059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:39.130059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:39.130109 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:28:39.130288 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:28:39.130289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:28:39.130289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:28:39.130390 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:28:39.130459 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.130707 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.130782 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:28:39.139204 17854 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:28:39.141556 17854 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:28:39.150347 18352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:43681 (local address 127.0.0.1:43566)
0929 03:28:39.145178 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.145206 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.145291 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.146348 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.146349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.146351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:28:39.146412 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.146414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.147802 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.147803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.149020 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.149022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.150064 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.150065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.150152 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.150164 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.150230 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.150275 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":15,"thread_start_us":89,"threads_started":1}
I20260929 03:28:39.150776 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43566 (local address 127.6.116.129:43681)
0929 03:28:39.144919 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.144923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.145279 (+ 356us) server_negotiation.cc:407] Connection header received
0929 03:28:39.145518 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.145523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.145545 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.145628 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:28:39.146505 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.147694 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.149120 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.149834 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.149979 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.150041 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.150668 (+ 627us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.150712 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260929 03:28:39.151592 18000 tablet_service.cc:1467] Processing CreateTablet for tablet 135f6d30fe8a4db2b406b128711d0d4a (DEFAULT_TABLE table=test-table [id=ea80e8170f144e5c8992c3fd750ca23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:28:39.151762 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:35751 (local address 127.0.0.1:35232)
0929 03:28:39.146014 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.146024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.146124 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.146416 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.146416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.146418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:28:39.146450 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.146451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.147719 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.147719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.148575 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.148577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.151545 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.151546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.151600 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.151604 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.151656 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.151694 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":73,"threads_started":1}
I20260929 03:28:39.152001 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f6d30fe8a4db2b406b128711d0d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:28:39.153031 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:38751 (local address 127.0.0.1:34486)
0929 03:28:39.145423 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.145433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.145474 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.145841 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.145844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.145846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:28:39.145902 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.145904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.150272 (+ 4368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.150273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.151445 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.151448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.152800 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.152801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.152867 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.152871 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.152927 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.152964 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":79,"threads_started":1}
I20260929 03:28:39.153321 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34486 (local address 127.6.116.130:38751)
0929 03:28:39.145606 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.145610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.145620 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:28:39.145658 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.145660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.145675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.145748 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:28:39.148729 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.150169 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.151877 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.152626 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.152702 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.152764 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.153193 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.153231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":66,"threads_started":1}
I20260929 03:28:39.153894 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 135f6d30fe8a4db2b406b128711d0d4a (DEFAULT_TABLE table=test-table [id=ea80e8170f144e5c8992c3fd750ca23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:28:39.154199 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f6d30fe8a4db2b406b128711d0d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:28:39.158833 18357 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: Bootstrap starting.
I20260929 03:28:39.158876 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.144654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.6.116.131:35751)
0929 03:28:39.145106 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.145110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.146095 (+ 985us) server_negotiation.cc:407] Connection header received
0929 03:28:39.146221 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.146223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.146240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.146309 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:28:39.146534 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.147607 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.148791 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.150309 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.150373 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.150434 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.158738 (+ 8304us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.158789 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":104,"threads_started":1}
I20260929 03:28:39.159082 18356 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Bootstrap starting.
I20260929 03:28:39.159603 18272 tablet_service.cc:1467] Processing CreateTablet for tablet 135f6d30fe8a4db2b406b128711d0d4a (DEFAULT_TABLE table=test-table [id=ea80e8170f144e5c8992c3fd750ca23c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:28:39.159896 18272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135f6d30fe8a4db2b406b128711d0d4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:28:39.160111 18356 tablet_bootstrap.cc:654] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Neither blocks nor log segments found. Creating new log.
I20260929 03:28:39.160151 18357 tablet_bootstrap.cc:654] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: Neither blocks nor log segments found. Creating new log.
I20260929 03:28:39.160444 18356 log.cc:824] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Log is configured to *not* fsync() on all Append() calls
I20260929 03:28:39.160442 18357 log.cc:824] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:28:39.161216 18356 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: No bootstrap required, opened a new log
I20260929 03:28:39.161229 18357 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: No bootstrap required, opened a new log
I20260929 03:28:39.161279 18357 ts_tablet_manager.cc:1397] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 03:28:39.161279 18356 ts_tablet_manager.cc:1397] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:28:39.163689 18356 raft_consensus.cc:348] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.163925 18356 raft_consensus.cc:374] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:28:39.164004 18356 raft_consensus.cc:729] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115b801c7a194581aae6551930310b38, State: Initialized, Role: FOLLOWER
I20260929 03:28:39.164175 18356 consensus_queue.cc:260] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.164496 18356 ts_tablet_manager.cc:1428] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 03:28:39.164672 18203 heartbeater.cc:503] Master 127.6.116.190:39189 was elected leader, sending a full tablet report...
I20260929 03:28:39.164706 18357 raft_consensus.cc:348] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.164870 18360 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Bootstrap starting.
I20260929 03:28:39.164903 18357 raft_consensus.cc:374] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:28:39.164954 18357 raft_consensus.cc:729] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017c23186a8a42d2a295d16bfd49374c, State: Initialized, Role: FOLLOWER
I20260929 03:28:39.165098 18357 consensus_queue.cc:260] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.165485 18357 ts_tablet_manager.cc:1428] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260929 03:28:39.165663 18066 heartbeater.cc:503] Master 127.6.116.190:39189 was elected leader, sending a full tablet report...
I20260929 03:28:39.165901 18360 tablet_bootstrap.cc:654] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Neither blocks nor log segments found. Creating new log.
I20260929 03:28:39.166215 18360 log.cc:824] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:28:39.169752 18360 tablet_bootstrap.cc:492] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: No bootstrap required, opened a new log
I20260929 03:28:39.169833 18360 ts_tablet_manager.cc:1397] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:28:39.171743 18360 raft_consensus.cc:348] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.171911 18360 raft_consensus.cc:374] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:28:39.171954 18360 raft_consensus.cc:729] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d759088772794b8a90ebf62b1c8eb4b8, State: Initialized, Role: FOLLOWER
I20260929 03:28:39.172065 18360 consensus_queue.cc:260] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.172292 18360 ts_tablet_manager.cc:1428] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:28:39.172465 18338 heartbeater.cc:503] Master 127.6.116.190:39189 was elected leader, sending a full tablet report...
I20260929 03:28:39.265988 18364 raft_consensus.cc:482] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:28:39.266155 18364 raft_consensus.cc:504] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.266556 18364 leader_election.cc:288] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017c23186a8a42d2a295d16bfd49374c (127.6.116.129:43681), 115b801c7a194581aae6551930310b38 (127.6.116.130:38751)
I20260929 03:28:39.270361 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.266690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:43681 (local address 127.6.116.131:40027)
0929 03:28:39.266740 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.266752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.266790 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.267099 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.267100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.267101 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:28:39.267170 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.267173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.268159 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.268160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.269339 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.269341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.270119 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.270120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.270192 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.270195 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.270239 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.270290 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":41}
I20260929 03:28:39.270370 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.267067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51997 (local address 127.6.116.130:38751)
0929 03:28:39.267107 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.267109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.267219 (+ 110us) server_negotiation.cc:407] Connection header received
0929 03:28:39.267312 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.267313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.267324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.267353 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:28:39.267547 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.268248 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.269522 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.270030 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.270071 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.270080 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.270281 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.270315 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:28:39.270582 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.266821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:38751 (local address 127.6.116.131:51997)
0929 03:28:39.267168 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.267178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:28:39.267241 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:28:39.267408 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:28:39.267409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:28:39.267410 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:28:39.267474 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.267475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.268371 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.268372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.269431 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:28:39.269433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.270131 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.270132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:28:39.270192 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.270195 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:28:39.270458 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 03:28:39.270501 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":97,"threads_started":1}
I20260929 03:28:39.270919 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f6d30fe8a4db2b406b128711d0d4a" candidate_uuid: "d759088772794b8a90ebf62b1c8eb4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "115b801c7a194581aae6551930310b38" is_pre_election: true
I20260929 03:28:39.271011 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.266889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40027 (local address 127.6.116.129:43681)
0929 03:28:39.266930 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.266932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.266940 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:28:39.266965 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.266966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.266976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.267026 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:28:39.267253 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.268041 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.269461 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.270020 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.270083 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.270090 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.270913 (+ 823us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.270955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:28:39.271088 18151 raft_consensus.cc:2454] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d759088772794b8a90ebf62b1c8eb4b8 in term 0.
I20260929 03:28:39.271246 18020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f6d30fe8a4db2b406b128711d0d4a" candidate_uuid: "d759088772794b8a90ebf62b1c8eb4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017c23186a8a42d2a295d16bfd49374c" is_pre_election: true
I20260929 03:28:39.271306 18228 leader_election.cc:302] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115b801c7a194581aae6551930310b38, d759088772794b8a90ebf62b1c8eb4b8; no voters:
I20260929 03:28:39.271363 18020 raft_consensus.cc:2454] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d759088772794b8a90ebf62b1c8eb4b8 in term 0.
I20260929 03:28:39.271407 18364 raft_consensus.cc:2781] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:28:39.271447 18364 raft_consensus.cc:482] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:28:39.271476 18364 raft_consensus.cc:3037] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:28:39.272291 18364 raft_consensus.cc:504] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.272415 18364 leader_election.cc:288] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [CANDIDATE]: Term 1 election: Requested vote from peers 017c23186a8a42d2a295d16bfd49374c (127.6.116.129:43681), 115b801c7a194581aae6551930310b38 (127.6.116.130:38751)
I20260929 03:28:39.272569 18020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f6d30fe8a4db2b406b128711d0d4a" candidate_uuid: "d759088772794b8a90ebf62b1c8eb4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017c23186a8a42d2a295d16bfd49374c"
I20260929 03:28:39.272603 18151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135f6d30fe8a4db2b406b128711d0d4a" candidate_uuid: "d759088772794b8a90ebf62b1c8eb4b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "115b801c7a194581aae6551930310b38"
I20260929 03:28:39.272642 18020 raft_consensus.cc:3037] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:28:39.272667 18151 raft_consensus.cc:3037] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:28:39.273461 18151 raft_consensus.cc:2454] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d759088772794b8a90ebf62b1c8eb4b8 in term 1.
I20260929 03:28:39.273488 18020 raft_consensus.cc:2454] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d759088772794b8a90ebf62b1c8eb4b8 in term 1.
I20260929 03:28:39.273620 18228 leader_election.cc:302] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 115b801c7a194581aae6551930310b38, d759088772794b8a90ebf62b1c8eb4b8; no voters:
I20260929 03:28:39.273680 18364 raft_consensus.cc:2781] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:28:39.273801 18364 raft_consensus.cc:686] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 1 LEADER]: Becoming Leader. State: Replica: d759088772794b8a90ebf62b1c8eb4b8, State: Running, Role: LEADER
I20260929 03:28:39.273882 18364 consensus_queue.cc:237] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } }
I20260929 03:28:39.274588 17854 catalog_manager.cc:5495] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d759088772794b8a90ebf62b1c8eb4b8 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "d759088772794b8a90ebf62b1c8eb4b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d759088772794b8a90ebf62b1c8eb4b8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260929 03:28:39.302424 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.299214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.6.116.131:35751)
0929 03:28:39.299264 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:28:39.299266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:28:39.299277 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:28:39.299348 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:28:39.299349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:28:39.299368 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:28:39.299497 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:28:39.299662 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.300424 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.301546 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:28:39.301849 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:28:39.301923 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:28:39.302075 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:28:39.302195 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:28:39.302281 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:28:39.302321 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:28:39.302340 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:28:39.310128 18151 raft_consensus.cc:1261] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 1 FOLLOWER]: Refusing update from remote peer d759088772794b8a90ebf62b1c8eb4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:28:39.310503 18364 consensus_queue.cc:1035] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "115b801c7a194581aae6551930310b38" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:28:39.312134 18020 raft_consensus.cc:1261] T 135f6d30fe8a4db2b406b128711d0d4a P 017c23186a8a42d2a295d16bfd49374c [term 1 FOLLOWER]: Refusing update from remote peer d759088772794b8a90ebf62b1c8eb4b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:28:39.312410 18364 consensus_queue.cc:1035] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017c23186a8a42d2a295d16bfd49374c" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 03:28:39 dist-test-slave-z2zg krb5kdc[17378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652518, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:28:39.318347 18369 mvcc.cc:205] Tried to move back new op lower bound from 7334512719091707904 to 7334512718946365440. Current Snapshot: MvccSnapshot[applied={T|T < 7334512719091707904}]
I20260929 03:28:39.320547 18372 mvcc.cc:205] Tried to move back new op lower bound from 7334512719091707904 to 7334512718946365440. Current Snapshot: MvccSnapshot[applied={T|T < 7334512719091707904}]
I20260929 03:28:39.323105 18370 mvcc.cc:205] Tried to move back new op lower bound from 7334512719091707904 to 7334512718946365440. Current Snapshot: MvccSnapshot[applied={T|T < 7334512719091707904}]
I20260929 03:28:39.324544 17854 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:28:39.324637 17854 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:28:39.325415 17854 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3725bb7e01244f8b966da3227b019b5a: Sending DeleteTablet for 3 replicas of tablet 135f6d30fe8a4db2b406b128711d0d4a
I20260929 03:28:39.325755 18133 tablet_service.cc:1514] Processing DeleteTablet for tablet 135f6d30fe8a4db2b406b128711d0d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:28:39 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:34486
I20260929 03:28:39.325963 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 17935
I20260929 03:28:39.326965 18272 tablet_service.cc:1514] Processing DeleteTablet for tablet 135f6d30fe8a4db2b406b128711d0d4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:28:39 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:35232
I20260929 03:28:39.327620 18380 tablet_replica.cc:330] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: stopping tablet replica
I20260929 03:28:39.327746 18380 raft_consensus.cc:2229] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:28:39.327913 18380 raft_consensus.cc:2258] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:28:39.328351 18380 ts_tablet_manager.cc:1905] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:28:39.330119 18379 tablet_replica.cc:330] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: stopping tablet replica
I20260929 03:28:39.330260 18379 raft_consensus.cc:2229] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:28:39.330406 18379 raft_consensus.cc:2258] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:28:39.330919 18379 ts_tablet_manager.cc:1905] T 135f6d30fe8a4db2b406b128711d0d4a P 115b801c7a194581aae6551930310b38: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:28:39.332630 17840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:43681 (error 108)
I20260929 03:28:39.332880 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 18072
I20260929 03:28:39.333607 18380 ts_tablet_manager.cc:1918] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:28:39.333735 18380 log.cc:1196] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/135f6d30fe8a4db2b406b128711d0d4a
I20260929 03:28:39.333735 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.333626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:43681 (local address 127.0.0.1:43576)
0929 03:28:39.333671 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.333703 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:28:39.333838 17840 catalog_manager.cc:4584] TS 017c23186a8a42d2a295d16bfd49374c (127.6.116.129:43681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 135f6d30fe8a4db2b406b128711d0d4a: Network error: Client connection negotiation failed: client connection to 127.6.116.129:43681: connect: Connection refused (error 111)
I20260929 03:28:39.338718 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:28:39.338617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:38751 (local address 127.0.0.1:34496)
0929 03:28:39.338660 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:28:39.338682 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:38751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 03:28:39.338816 18380 ts_tablet_manager.cc:1939] T 135f6d30fe8a4db2b406b128711d0d4a P d759088772794b8a90ebf62b1c8eb4b8: Deleting consensus metadata
I20260929 03:28:39.339450 17840 catalog_manager.cc:4841] TS d759088772794b8a90ebf62b1c8eb4b8 (127.6.116.131:35751): tablet 135f6d30fe8a4db2b406b128711d0d4a (table test-table [id=ea80e8170f144e5c8992c3fd750ca23c]) successfully deleted
I20260929 03:28:39.339512 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 18209
I20260929 03:28:39.345113 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 17822
I20260929 03:28:39.351677 6610 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:28:39.696523 6610 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:28:39.696625 6610 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:28:40.031437 6610 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:28:40.031 UTC [17408] LOG: received smart shutdown request
2026-09-29 03:28:40.034 UTC [17408] LOG: background worker "logical replication launcher" (PID 17417) exited with exit code 1
2026-09-29 03:28:40.034 UTC [17412] LOG: shutting down
2026-09-29 03:28:40.181 UTC [17408] LOG: database system is shut down
2026-09-29T03:28:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44972 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:28:40 dist-test-slave-z2zg krb5kdc[18390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:28:40 dist-test-slave-z2zg krb5kdc[18390](info): set up 2 sockets
Sep 29 03:28:40 dist-test-slave-z2zg krb5kdc[18390](info): commencing operation
krb5kdc: starting...
W20260929 03:28:41.276705 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:28:41 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:28:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangeradmin_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangeradmin_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangerlookup_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangerlookup_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
I20260929 03:28:41.422708 6610 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFuIZmB/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/postgres -l logfile start
2026-09-29 03:28:43.763 UTC [18420] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 03:28:43.763 UTC [18420] LOG: listening on IPv4 address "127.6.116.148", port 50947
2026-09-29 03:28:43.767 UTC [18420] LOG: listening on Unix socket "/tmp/.s.PGSQL.50947"
2026-09-29 03:28:43.800 UTC [18423] LOG: database system was shut down at 2026-09-29 03:28:42 UTC
2026-09-29 03:28:43.803 UTC [18420] LOG: database system is ready to accept connections
I20260929 03:28:44.732645 6610 mini_postgres.cc:96] Postgres bound to 50947
2026-09-29 03:28:44.738 UTC [18431] FATAL: database "slave" does not exist
127.6.116.148:50947 - accepting connections
I20260929 03:28:44.739033 6610 mini_ranger.cc:161] Starting Ranger...
I20260929 03:28:44.753217 6610 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 03:28:45.205390 6610 mini_ranger.cc:87] Created ranger Postgres database
I20260929 03:28:45.205487 6610 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin
2026-09-29 03:28:45,489 [I] DB FLAVOR :POSTGRES
2026-09-29 03:28:45,489 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:28:45,489 [I] Checking connection..
2026-09-29 03:28:45,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 03:28:45,729 [I] Checking connection passed.
2026-09-29 03:28:45,729 [I] --------- Verifying version history table ---------
2026-09-29 03:28:45,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:45,982 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:28:45,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:46,204 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 03:28:46,205 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 03:28:46,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 03:28:46.438 UTC [18495] WARNING: there is no transaction in progress
2026-09-29 03:28:46,444 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 03:28:46,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 03:28:46,682 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 03:28:46,682 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:28:46,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 03:28:46,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 03:28:47,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z2zg.c.gcp-upstream.internal','N') ;"
2026-09-29 03:28:47,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:28:47,618 [I] Table x_portal_user does not exist in database ranger
2026-09-29 03:28:47,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:28:47,837 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 03:28:47,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:28:48,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:28:48,307 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 03:28:48,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 03:28:49.164 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.178 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.193 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.206 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.220 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.235 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.285 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.290 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.302 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.311 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.320 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.332 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.341 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.347 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.352 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49.358 UTC [18603] WARNING: there is no transaction in progress
2026-09-29 03:28:49,938 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 03:28:49,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 03:28:50,155 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 03:28:50,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 03:28:50,404 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 03:28:50,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 03:28:50,646 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 03:28:50,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 03:28:50,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 03:28:51,125 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z2zg.c.gcp-upstream.internal';"
2026-09-29 03:28:51,375 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 03:28:51,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 03:28:51,622 [I] DB_PATCHES have already been applied
I20260929 03:28:51.627851 6610 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFuIZmB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 03:28:51.627921 6610 mini_ranger.cc:193] Using host: 127.6.116.148
I20260929 03:28:51.630895 6610 mini_ranger.cc:227] Ranger admin URL: http://127.6.116.148:34921
Sep 29, 2026 3:28:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:28:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:28:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 03:28:52.732386 6610 mini_ranger.cc:160] Time spent starting Ranger: real 7.993s user 0.003s sys 0.005s
Sep 29, 2026 3:29:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:29:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 29, 2026 3:29:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:29:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:29:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 03:29:13.164533 6610 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.116.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.116.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangerkms_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.116.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/rangerkms_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.116.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.116.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.116.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/HTTP_127.6.116.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 03:29:13 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 03:29:13.237809 6610 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 03:29:13.245745 6610 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 03:29:14.055508 6610 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 03:29:14.055591 6610 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms
I20260929 03:29:14.055613 6610 mini_ranger_kms.cc:227] Using postgres at 127.6.116.148:50947
2026-09-29 03:29:14,216 [I] DB FLAVOR :POSTGRES
2026-09-29 03:29:14,216 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 03:29:14,216 [I] Checking connection
2026-09-29 03:29:14,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 03:29:14,449 [I] connection success
2026-09-29 03:29:14,449 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 03:29:14,449 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 03:29:14,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 03:29:14,671 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 03:29:14,671 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 03:29:14,671 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 03:29:14,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.116.148:50947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 03:29:14,923 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 03:29:15.336750 6610 mini_ranger_kms.cc:310] Created kms service
I20260929 03:29:15.487152 6610 mini_ranger_kms.cc:326] Created kudu user
I20260929 03:29:15.526269 6610 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 03:29:15 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652553, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:15.886103 6610 mini_ranger_kms.cc:384] Added ranger policy
I20260929 03:29:15.886337 6610 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFuIZmB/build/release/bin/postgresql.jar:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFuIZmB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 03:29:15.886377 6610 mini_ranger_kms.cc:240] Using host: 127.6.116.148
I20260929 03:29:15.889981 6610 mini_ranger_kms.cc:276] Ranger KMS PID: 18783
I20260929 03:29:15.890049 6610 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.116.148:48975
Sep 29, 2026 3:29:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:29:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:29:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:29:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:29:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:29:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 03:29:18.060727 6610 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.823s user 0.000s sys 0.143s
I20260929 03:29:18.060824 6610 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 03:29:18.060871 6610 mini_ranger_kms.cc:401] 127.6.116.148:48975/kms/v1/keys
Sep 29 03:29:18 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 03:29:18 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
Sep 29 03:29:18 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652558, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.116.148@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
Sep 29 03:29:20 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:29:21.020071 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39731
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:39731
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:21.097828 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:21.097998 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:21.098017 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:21.099078 18834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:21.099113 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:21.099126 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:21.099189 18834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:21.099206 18834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:21.100668 18834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.116.148:48975/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:39731
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:21.100822 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:21.101048 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:21.105677 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:21.105698 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:21.105751 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:21.105768 18834 server_base.cc:1034] running on GCE node
I20260929 03:29:21.106205 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:21.106493 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:21.107654 18834 hybrid_clock.cc:648] HybridClock initialized: now 1790652561107631 us; error 35 us; skew 500 ppm
Sep 29 03:29:21 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:21.113935 18834 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:21.114322 18834 webserver.cc:460] Webserver started at http://127.6.116.190:44759/ using document root <none> and password file <none>
I20260929 03:29:21.114547 18834 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:21.114605 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:21.114754 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:29:21 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:21.159107 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "8f0e2cef9cd343f1b577631920ce95ee"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "b77397642f1e0add1fecaa3ad2649049"
server_key_iv: "6b66cd8f50b2338529e672c1f8d92936"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:21.159547 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "8f0e2cef9cd343f1b577631920ce95ee"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "b77397642f1e0add1fecaa3ad2649049"
server_key_iv: "6b66cd8f50b2338529e672c1f8d92936"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:21.160954 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:29:21.172020 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:21.172202 18834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:21.172290 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "8f0e2cef9cd343f1b577631920ce95ee"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "b77397642f1e0add1fecaa3ad2649049"
server_key_iv: "6b66cd8f50b2338529e672c1f8d92936"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:21.172365 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:21.188092 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:21.189818 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:21.189975 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:21.194535 18834 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:39731
I20260929 03:29:21.194604 18901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:39731 every 8 connection(s)
I20260929 03:29:21.194949 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:21.195468 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 18834
I20260929 03:29:21.195561 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:29:21.195726 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:21.195761 6610 external_mini_cluster.cc:1428] Setting key 9d59bd4e053420f735c68010f84eba63
I20260929 03:29:21.198763 18902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Bootstrap starting.
I20260929 03:29:21.199425 18902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:21.199707 18902 log.cc:824] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:21.200363 18902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: No bootstrap required, opened a new log
I20260929 03:29:21.202395 18902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:21.202564 18902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:21.202629 18902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e2cef9cd343f1b577631920ce95ee, State: Initialized, Role: FOLLOWER
Sep 29 03:29:21 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652560, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:21.203480 18902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:21.203584 18902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:21.203640 18902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:21.203711 18902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:21.204393 18902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:21.204509 18902 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0e2cef9cd343f1b577631920ce95ee; no voters:
I20260929 03:29:21.204658 18902 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:29:21.204831 18908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:21.204921 18902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:21.205736 18908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 LEADER]: Becoming Leader. State: Replica: 8f0e2cef9cd343f1b577631920ce95ee, State: Running, Role: LEADER
I20260929 03:29:21.205873 18908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:21.207242 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:21.196316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.6.116.190:39731)
0929 03:29:21.196451 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:21.196460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:21.196486 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:29:21.197218 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:21.197227 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:21.197298 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:21.197439 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:21.198149 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:21.199476 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:21.200514 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:21.201088 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:21.201214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:21.203841 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:21.203845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:21.203846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:21.203856 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:21.205539 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:21.205957 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:21.205960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:21.205960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:21.206061 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:21.206259 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:21.206260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:21.206260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:21.206581 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:21.206761 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:21.206853 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:29:21.206961 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260929 03:29:21.207643 18902 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:29:21.207906 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0e2cef9cd343f1b577631920ce95ee. Latest consensus state: current_term: 1 leader_uuid: "8f0e2cef9cd343f1b577631920ce95ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } } }
I20260929 03:29:21.207996 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:21.209831 18907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0e2cef9cd343f1b577631920ce95ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } } }
I20260929 03:29:21.210085 18907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:21.210352 18922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:21.211285 18922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:21.213173 18922 catalog_manager.cc:1347] Generated new cluster ID: 39139476998e4ba3b22daed55a8ef6f0
I20260929 03:29:21.213263 18922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:29:21.227965 18922 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:29:21.228693 18922 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:21.236927 18922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Generated new TSK 0
I20260929 03:29:21.237136 18922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:29:21.854131 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:21.966279 18947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:21.966549 18947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:21.966583 18947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:21.968182 18947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:21.968243 18947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:21.968286 18947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:21.970335 18947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:21.970551 18947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:21.970806 18947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:21.976810 18952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:21.976822 18955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:21.977042 18953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:21.977514 18947 server_base.cc:1034] running on GCE node
I20260929 03:29:21.977705 18947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:21.977965 18947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:21.979122 18947 hybrid_clock.cc:648] HybridClock initialized: now 1790652561979109 us; error 40 us; skew 500 ppm
Sep 29 03:29:21 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:21.981894 18947 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:21.982267 18947 webserver.cc:460] Webserver started at http://127.6.116.129:34931/ using document root <none> and password file <none>
I20260929 03:29:21.982499 18947 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:21.982559 18947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:21.982692 18947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:29:21 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:21.996541 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "5291f3066cad448fbec0fd6991b76ec0"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "8e857e91de3edb0556f05db7e7f42de2"
server_key_iv: "5de9b80dbdb7d3a9c1ca347a97a797a1"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:21.997010 18947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "5291f3066cad448fbec0fd6991b76ec0"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "8e857e91de3edb0556f05db7e7f42de2"
server_key_iv: "5de9b80dbdb7d3a9c1ca347a97a797a1"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:21.998724 18947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:22.011802 18962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.011993 18947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:29:22.012069 18947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "5291f3066cad448fbec0fd6991b76ec0"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "8e857e91de3edb0556f05db7e7f42de2"
server_key_iv: "5de9b80dbdb7d3a9c1ca347a97a797a1"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.012136 18947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:22.034230 18947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:22.036698 18947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:22.036880 18947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:22.037111 18947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:22.037487 18947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:22.037565 18947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.037650 18947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:22.037712 18947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.044498 18947 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:40671
I20260929 03:29:22.044849 18947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:29:22.045849 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:40671 every 8 connection(s)
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652561, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:22.052397 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 18947
I20260929 03:29:22.052498 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:29:22.052678 6610 external_mini_cluster.cc:1428] Setting key a4af54bbf414f12f7cda779dcdde07c8
I20260929 03:29:22.056353 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.045583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.129:48305)
0929 03:29:22.046063 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:22.046103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:22.046605 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:22.046875 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:22.046879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:22.047126 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:22.047430 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.047435 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.048127 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.048128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.048943 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.048945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.049385 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.049386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.050411 (+ 1025us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.050931 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:22.050940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:22.052754 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:22.054774 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.054780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.054782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.055077 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.055310 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.055311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.055311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.055428 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.055702 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:22.055707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:22.055850 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 03:29:22.055943 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 03:29:22.056081 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":79,"threads_started":1}
I20260929 03:29:22.057015 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.045892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:48305 (local address 127.6.116.190:39731)
0929 03:29:22.046040 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:22.046043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:22.046196 (+ 153us) server_negotiation.cc:407] Connection header received
0929 03:29:22.046718 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:22.046720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:22.046735 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:22.046810 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:22.047521 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.048024 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.049034 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.049288 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.049356 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.052858 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:22.052859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:22.052860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:22.052867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:22.054682 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.055161 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.055162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.055162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.055236 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.055495 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.055496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.055496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.055624 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:22.055679 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:22.056829 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0929 03:29:22.056912 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260929 03:29:22.057641 19078 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:22.057732 19078 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.057925 19078 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:22.058501 18866 ts_manager.cc:194] Registered new tserver with Master: 5291f3066cad448fbec0fd6991b76ec0 (127.6.116.129:40671)
I20260929 03:29:22.059405 18866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:48305
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:29:22.082345 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:22.183892 19084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:22.184128 19084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:22.184166 19084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:22.185259 19084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:22.185326 19084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:22.185371 19084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:22.186803 19084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:22.187033 19084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:22.187287 19084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:22.192035 19092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.192267 19094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.192495 19091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:22.192458 19084 server_base.cc:1034] running on GCE node
I20260929 03:29:22.192716 19084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:22.192947 19084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:22.194149 19084 hybrid_clock.cc:648] HybridClock initialized: now 1790652562194123 us; error 32 us; skew 500 ppm
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:22.197424 19084 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:22.197839 19084 webserver.cc:460] Webserver started at http://127.6.116.130:46223/ using document root <none> and password file <none>
I20260929 03:29:22.198128 19084 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:22.198266 19084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:22.198423 19084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:22.212198 19084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "4378c938b8cd4a928ee92fc59ad8e2d9"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "e958ed72b89d3db9f4013e8fc4f137f1"
server_key_iv: "323bd879158ed1aff3d8c54a6492c642"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.212987 19084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "4378c938b8cd4a928ee92fc59ad8e2d9"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "e958ed72b89d3db9f4013e8fc4f137f1"
server_key_iv: "323bd879158ed1aff3d8c54a6492c642"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.219477 19084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:22.239136 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.239341 19084 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260929 03:29:22.239420 19084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "4378c938b8cd4a928ee92fc59ad8e2d9"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "e958ed72b89d3db9f4013e8fc4f137f1"
server_key_iv: "323bd879158ed1aff3d8c54a6492c642"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.239487 19084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:22.274034 19084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:22.276368 19084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:22.276540 19084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:22.276742 19084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:22.277099 19084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:22.277171 19084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.277248 19084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:22.277303 19084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.283684 19084 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45489
I20260929 03:29:22.283751 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45489 every 8 connection(s)
I20260929 03:29:22.284194 19084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:29:22.288393 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19084
I20260929 03:29:22.288482 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:29:22.288659 6610 external_mini_cluster.cc:1428] Setting key c372c75892b71793de2b14a5eedb1ddb
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:22.297967 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.284943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.130:39785)
0929 03:29:22.285143 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:22.285201 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:22.285851 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:22.286159 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:22.286163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:22.286396 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:22.286644 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.286650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.287633 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.287634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.288905 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.288908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.289479 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.289479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.289611 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.290118 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:22.290127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:22.294322 (+ 4195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:22.296305 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.296312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.296314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.296604 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.296845 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.296846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.296847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.296978 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.297313 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:22.297319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:22.297472 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 03:29:22.297598 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:29:22.297747 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260929 03:29:22.298682 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.285244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:39785 (local address 127.6.116.190:39731)
0929 03:29:22.285294 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:22.285296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:22.285309 (+ 13us) server_negotiation.cc:407] Connection header received
0929 03:29:22.285991 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:22.285992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:22.286005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:22.286077 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:22.286757 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.287527 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.289012 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.289328 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.289374 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.294424 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:22.294426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:22.294426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:22.294434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:22.296230 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.296690 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.296692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.296692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.296761 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.297055 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.297056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.297056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.297203 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:22.297282 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:22.298496 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0929 03:29:22.298590 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:29:22.299223 19215 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:22.299311 19215 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.299481 19215 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:22.299803 18866 ts_manager.cc:194] Registered new tserver with Master: 4378c938b8cd4a928ee92fc59ad8e2d9 (127.6.116.130:45489)
I20260929 03:29:22.300287 18866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:39785
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:29:22.321702 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:22.438063 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:22.438323 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:22.438361 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:22.440028 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:22.440090 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:22.440130 19221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:22.442309 19221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:22.442525 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:22.442828 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:22.448635 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:22.449120 19221 server_base.cc:1034] running on GCE node
W20260929 03:29:22.449311 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.449425 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:22.449669 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:22.449887 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:22.451056 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790652562451022 us; error 31 us; skew 500 ppm
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:22.455318 19221 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:22.455838 19221 webserver.cc:460] Webserver started at http://127.6.116.131:46567/ using document root <none> and password file <none>
I20260929 03:29:22.456100 19221 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:22.456154 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:22.456280 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:22.471690 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "7d965e6ca2b243ce8573a02a3340572d"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "ae19340483bf4fdd4301b4a2bb87fa46"
server_key_iv: "888f9f41ddc17a06e528d3d8b432151c"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.472098 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "7d965e6ca2b243ce8573a02a3340572d"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "ae19340483bf4fdd4301b4a2bb87fa46"
server_key_iv: "888f9f41ddc17a06e528d3d8b432151c"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.483464 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260929 03:29:22.495736 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.495954 19221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:22.496080 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "7d965e6ca2b243ce8573a02a3340572d"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "ae19340483bf4fdd4301b4a2bb87fa46"
server_key_iv: "888f9f41ddc17a06e528d3d8b432151c"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.496218 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:22.518836 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:22.521313 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:22.521492 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:22.521781 19221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:22.522184 19221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:22.522267 19221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.522343 19221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:22.522406 19221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.529697 19221 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:45417
I20260929 03:29:22.529808 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:45417 every 8 connection(s)
I20260929 03:29:22.530177 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:29:22.531028 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19221
I20260929 03:29:22.531152 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:29:22.531342 6610 external_mini_cluster.cc:1428] Setting key 84331e2ea99565f7692b9e8891add06c
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:22.541085 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.530933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39129 (local address 127.6.116.190:39731)
0929 03:29:22.530994 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:22.530996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:22.531214 (+ 218us) server_negotiation.cc:407] Connection header received
0929 03:29:22.531928 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:22.531929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:22.531942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:22.531974 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:22.532812 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.533569 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.535088 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.535402 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.535450 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.537915 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:22.537917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:22.537917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:22.537924 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:22.539609 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.540079 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.540080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.540081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.540136 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.540392 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.540392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.540393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.540516 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:22.540554 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:22.540910 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 03:29:22.540990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:29:22.541534 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.530857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.131:39129)
0929 03:29:22.531091 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:22.531133 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:22.531796 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:22.532158 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:22.532162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:22.532396 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:22.532687 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.532694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.533682 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.533683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.534951 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.534954 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.535553 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.535554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.535687 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.536217 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:22.536227 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:22.537807 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:22.539702 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.539708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.539711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.539992 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.540203 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.540203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.540204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.540319 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.540595 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:22.540600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:22.540756 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 03:29:22.541176 (+ 420us) client_negotiation.cc:240] Negotiation successful
0929 03:29:22.541324 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260929 03:29:22.542227 19350 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:22.542330 19350 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.542491 19350 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:22.543143 18865 ts_manager.cc:194] Registered new tserver with Master: 7d965e6ca2b243ce8573a02a3340572d (127.6.116.131:45417)
I20260929 03:29:22.543613 18865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:39129
I20260929 03:29:22.553879 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:29:22.554549 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 18947
I20260929 03:29:22.560428 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19084
I20260929 03:29:22.565690 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19221
I20260929 03:29:22.569854 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 18834
I20260929 03:29:22.575587 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39731
--webserver_interface=127.6.116.190
--webserver_port=44759
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:39731
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:22.652494 19353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:22.652669 19353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:22.652690 19353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:22.653779 19353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:22.653818 19353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:22.653832 19353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:22.653869 19353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:22.653896 19353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:22.655483 19353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:39731
--ranger_config_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=44759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:22.655702 19353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:22.655923 19353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:22.660290 19361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.660277 19359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.660518 19358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:22.660508 19353 server_base.cc:1034] running on GCE node
I20260929 03:29:22.660733 19353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:22.660955 19353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:22.662107 19353 hybrid_clock.cc:648] HybridClock initialized: now 1790652562662098 us; error 35 us; skew 500 ppm
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:22.664424 19353 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:22.664855 19353 webserver.cc:460] Webserver started at http://127.6.116.190:44759/ using document root <none> and password file <none>
I20260929 03:29:22.665069 19353 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:22.665139 19353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:22.720454 19353 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:22.721144 19369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.721314 19353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:22.721436 19353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "8f0e2cef9cd343f1b577631920ce95ee"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "b77397642f1e0add1fecaa3ad2649049"
server_key_iv: "6b66cd8f50b2338529e672c1f8d92936"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.721764 19353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:22.740746 19353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:22.742189 19353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:22.742354 19353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:22.746635 19353 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:39731
I20260929 03:29:22.746678 19421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:39731 every 8 connection(s)
I20260929 03:29:22.747020 19353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:22.748178 19422 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:29:22.748799 19422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Bootstrap starting.
I20260929 03:29:22.750640 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19353
I20260929 03:29:22.750968 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:40671
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:39731
--webserver_port=34931
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:22.757189 19422 log.cc:824] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:22.758862 19422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:29:22.759196 19422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Bootstrap complete.
I20260929 03:29:22.761639 19422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:22.761868 19422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e2cef9cd343f1b577631920ce95ee, State: Initialized, Role: FOLLOWER
I20260929 03:29:22.762032 19422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:22.762096 19422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:22.762161 19422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:22.762205 19422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:29:22.763263 19422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:22.763368 19422 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0e2cef9cd343f1b577631920ce95ee; no voters:
I20260929 03:29:22.763564 19422 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:29:22.763597 19426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:29:22.763698 19426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [term 2 LEADER]: Becoming Leader. State: Replica: 8f0e2cef9cd343f1b577631920ce95ee, State: Running, Role: LEADER
I20260929 03:29:22.763798 19426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } }
I20260929 03:29:22.763825 19422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:22.764070 19427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8f0e2cef9cd343f1b577631920ce95ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } } }
I20260929 03:29:22.764160 19427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:22.764070 19426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0e2cef9cd343f1b577631920ce95ee. Latest consensus state: current_term: 2 leader_uuid: "8f0e2cef9cd343f1b577631920ce95ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0e2cef9cd343f1b577631920ce95ee" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39731 } } }
I20260929 03:29:22.764361 19426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:22.764673 19430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:22.765177 19430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:22.765308 19430 catalog_manager.cc:1259] Loaded cluster ID: 39139476998e4ba3b22daed55a8ef6f0
I20260929 03:29:22.765352 19430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:29:22.765523 19422 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 03:29:22.768950 19430 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:22.769502 19430 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Loaded TSK: 0
I20260929 03:29:22.769802 19430 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:29:22.847954 19424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:22.848150 19424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:22.848188 19424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:22.849263 19424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:22.849324 19424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:22.849377 19424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:22.850744 19424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:40671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=34931
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:22.850982 19424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:22.851195 19424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:22.856993 19460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.857180 19462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:22.856986 19459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:22.857686 19424 server_base.cc:1034] running on GCE node
I20260929 03:29:22.857878 19424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:22.858105 19424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:22.861389 19424 hybrid_clock.cc:648] HybridClock initialized: now 1790652562861376 us; error 25 us; skew 500 ppm
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:22.863919 19424 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:22.864441 19424 webserver.cc:460] Webserver started at http://127.6.116.129:34931/ using document root <none> and password file <none>
I20260929 03:29:22.864670 19424 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:22.864750 19424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:22.883347 19424 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260929 03:29:22.884222 19470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.884383 19424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:22.884471 19424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "5291f3066cad448fbec0fd6991b76ec0"
format_stamp: "Formatted at 2026-09-29 03:29:21 on dist-test-slave-z2zg"
server_key: "8e857e91de3edb0556f05db7e7f42de2"
server_key_iv: "5de9b80dbdb7d3a9c1ca347a97a797a1"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:22.884820 19424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:22.900807 19424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:22.903077 19424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:22.903231 19424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:22.903460 19424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:22.903861 19424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:22.903923 19424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.903957 19424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:22.903976 19424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:22.910342 19424 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:40671
I20260929 03:29:22.910547 19583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:40671 every 8 connection(s)
I20260929 03:29:22.910779 19424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:29:22.917377 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19424
I20260929 03:29:22.917681 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:45489
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:39731
--webserver_port=46223
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:29:22 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652562, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:22.930451 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.911940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:49861 (local address 127.6.116.190:39731)
0929 03:29:22.912882 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:22.912888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:22.912910 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:29:22.913064 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:22.913068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:22.913110 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:22.913238 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:22.913733 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.914633 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.915812 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.916430 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.916529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.926902 (+ 10373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:22.926905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:22.926906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:22.926917 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:22.928738 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.929171 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.929174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.929174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.929273 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:22.929507 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:22.929508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:22.929508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:22.929747 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:22.929797 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:22.930131 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 03:29:22.930235 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":93,"threads_started":1}
I20260929 03:29:22.930941 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:22.911588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.129:49861)
0929 03:29:22.912182 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:22.912230 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:22.912727 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:22.913274 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:22.913278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:22.913462 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:22.913626 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.913630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.914743 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.914744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.915717 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:22.915720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:22.916556 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:22.916557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:22.916677 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:22.917613 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:22.917622 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:22.926773 (+ 9151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:22.928829 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.928835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.928837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.929090 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.929341 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:22.929341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:22.929342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:22.929444 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:22.929841 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:22.929846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:22.929969 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 03:29:22.930592 (+ 623us) client_negotiation.cc:240] Negotiation successful
0929 03:29:22.930730 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":81,"threads_started":1}
I20260929 03:29:22.931669 19584 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:22.931759 19584 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.931929 19584 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:22.932634 19584 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:39731 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:29:22.932700 19584 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.932757 19584 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:22.933017 19584 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:22.933087 19584 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:22.933357 19584 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:29:23.023928 19588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:23.024178 19588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:23.024219 19588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:23.026027 19588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:23.026095 19588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:23.026154 19588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:23.028446 19588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:45489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=46223
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:23.028707 19588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:23.029085 19588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:23.035145 19593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:23.035406 19596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:23.035796 19588 server_base.cc:1034] running on GCE node
W20260929 03:29:23.035830 19594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:23.036114 19588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:23.036367 19588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:23.037531 19588 hybrid_clock.cc:648] HybridClock initialized: now 1790652563037515 us; error 32 us; skew 500 ppm
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:23.040588 19588 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:23.041065 19588 webserver.cc:460] Webserver started at http://127.6.116.130:46223/ using document root <none> and password file <none>
I20260929 03:29:23.041260 19588 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:23.041314 19588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:23.071864 19588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260929 03:29:23.072806 19604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.073004 19588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:23.073081 19588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "4378c938b8cd4a928ee92fc59ad8e2d9"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "e958ed72b89d3db9f4013e8fc4f137f1"
server_key_iv: "323bd879158ed1aff3d8c54a6492c642"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:23.073513 19588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:23.108386 19588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:23.110348 19588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:23.110517 19588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:23.110770 19588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:23.111157 19588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:23.111219 19588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.111275 19588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:23.111304 19588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.118408 19588 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45489
I20260929 03:29:23.118500 19717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45489 every 8 connection(s)
I20260929 03:29:23.118819 19588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:29:23.119745 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19588
I20260929 03:29:23.120059 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.116.148:48975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:45417
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:39731
--webserver_port=46567
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:23.138455 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:23.119620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:40951 (local address 127.6.116.190:39731)
0929 03:29:23.119670 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:23.119672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:23.119939 (+ 267us) server_negotiation.cc:407] Connection header received
0929 03:29:23.120599 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:23.120601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:23.120617 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:23.120694 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:23.121407 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.122125 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.131170 (+ 9045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.131516 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.131590 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:23.135507 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:23.135509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:23.135509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:23.135517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:23.137088 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.137510 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.137511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.137512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.137571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.137822 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.137823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.137823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.137936 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:23.138017 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:23.138280 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 03:29:23.138369 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:29:23.138605 19720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:23.119604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.130:40951)
0929 03:29:23.119782 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:23.119828 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:23.120475 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:23.120782 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:23.120787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:23.121001 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:23.121263 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:23.121270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.122243 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.122243 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:23.123534 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:23.123536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.131726 (+ 8190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.131727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:23.131866 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:23.132402 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:23.132412 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:23.135381 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:23.137199 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:23.137206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:23.137207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:23.137416 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:23.137640 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:23.137641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:23.137641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:23.137750 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:23.138009 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:23.138015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:23.138143 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 03:29:23.138261 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:29:23.138389 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260929 03:29:23.139274 19718 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:23.139387 19718 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.139561 19718 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:23.140205 19718 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:39731 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:29:23.140290 19718 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.140364 19718 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:23.140668 19718 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.140749 19718 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:23.141048 19718 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:29:23.238320 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:23.238647 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:23.238727 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:23.240355 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:23.240458 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:23.240512 19721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:23.242532 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.116.148:48975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:45417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=46567
--tserver_master_addrs=127.6.116.190:39731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:23.242975 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:23.243261 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:23.248461 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:23.248528 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:23.248466 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:23.248746 19721 server_base.cc:1034] running on GCE node
I20260929 03:29:23.248940 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:23.249178 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:23.250331 19721 hybrid_clock.cc:648] HybridClock initialized: now 1790652563250313 us; error 35 us; skew 500 ppm
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:23.252624 19721 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:23.253156 19721 webserver.cc:460] Webserver started at http://127.6.116.131:46567/ using document root <none> and password file <none>
I20260929 03:29:23.253419 19721 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:23.253497 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:23.274449 19721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:23.275527 19738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.275735 19721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:23.275830 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "7d965e6ca2b243ce8573a02a3340572d"
format_stamp: "Formatted at 2026-09-29 03:29:22 on dist-test-slave-z2zg"
server_key: "ae19340483bf4fdd4301b4a2bb87fa46"
server_key_iv: "888f9f41ddc17a06e528d3d8b432151c"
server_key_version: "kuduclusterkey@0"
I20260929 03:29:23.276324 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:23.287189 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:23.291770 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:23.291937 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:23.292174 19721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:23.292510 19721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:23.292558 19721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.292595 19721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:23.292620 19721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:23.300035 19721 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:45417
I20260929 03:29:23.300065 19851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:45417 every 8 connection(s)
I20260929 03:29:23.300474 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:23.307564 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19721
I20260929 03:29:23.309894 19587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:23.301236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40245 (local address 127.6.116.190:39731)
0929 03:29:23.301309 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:23.301312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:23.301552 (+ 240us) server_negotiation.cc:407] Connection header received
0929 03:29:23.302042 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:23.302043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:23.302056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:23.302107 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:23.302967 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.303776 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.304761 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.305043 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.305121 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:23.306967 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:23.306968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:23.306968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:23.306976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:23.308440 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.308894 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.308894 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.308895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.308963 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.309235 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.309236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.309237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.309360 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:23.309400 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:23.309710 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:29:23.309790 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:29:23.310370 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:23.301198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39731 (local address 127.6.116.131:40245)
0929 03:29:23.301447 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:23.301475 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:23.301942 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:23.302190 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:23.302193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:23.302355 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:23.302575 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:23.302580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.303879 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.303879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:23.304672 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:23.304674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.305149 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.305150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:23.305244 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:23.305683 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:23.305690 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:23.306887 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:23.308512 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:23.308518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:23.308520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:23.308782 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:23.309033 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:23.309034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:23.309034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:23.309156 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:23.309442 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:23.309447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:23.309581 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:29:23.310023 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 03:29:23.310162 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":107,"threads_started":1}
I20260929 03:29:23.310932 19852 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39731
I20260929 03:29:23.311017 19852 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.311195 19852 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:23.311837 19852 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:39731 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 03:29:23.311913 19852 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.311975 19852 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:23.312286 19852 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.312350 19852 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
W20260929 03:29:23.312657 19852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:29:23.316639 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:23.307978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.6.116.190:39731)
0929 03:29:23.308094 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:23.308096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:23.308624 (+ 528us) server_negotiation.cc:407] Connection header received
0929 03:29:23.308702 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:23.308703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:23.308715 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:23.308791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:23.309894 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.310563 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.312203 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:23.312513 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:23.312569 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:23.313847 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:23.313848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:23.313849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:23.313856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:23.315482 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.315879 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.315879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.315880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.315932 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:23.316176 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:23.316177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:23.316177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:23.316307 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:23.316433 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:23.316518 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:29:23.316578 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
Sep 29 03:29:23 dist-test-slave-z2zg krb5kdc[18390](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:23.933547 19584 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:23.933696 19584 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:23.934083 19386 ts_manager.cc:194] Registered new tserver with Master: 5291f3066cad448fbec0fd6991b76ec0 (127.6.116.129:40671)
I20260929 03:29:23.935084 19386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:49861
I20260929 03:29:24.141223 19718 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:24.141356 19718 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:24.141593 19386 ts_manager.cc:194] Registered new tserver with Master: 4378c938b8cd4a928ee92fc59ad8e2d9 (127.6.116.130:45489)
I20260929 03:29:24.142143 19386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:40951
Sep 29 03:29:24 dist-test-slave-z2zg krb5kdc[18390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652563, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for HTTP/127.6.116.148@KRBTEST.COM
I20260929 03:29:24.312853 19852 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:24.312987 19852 heartbeater.cc:511] Master 127.6.116.190:39731 requested a full tablet report, sending...
I20260929 03:29:24.313314 19386 ts_manager.cc:194] Registered new tserver with Master: 7d965e6ca2b243ce8573a02a3340572d (127.6.116.131:45417)
I20260929 03:29:24.313812 19386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:40245
I20260929 03:29:24.319192 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:29:24.328585 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.320148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.6.116.190:39731)
0929 03:29:24.320456 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.320459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.320605 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:29:24.320627 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.320628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.320638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.320689 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:24.321783 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.322492 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.323477 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.323769 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.323839 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.325536 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:24.325537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:24.325537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:24.325544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:24.327340 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:24.327701 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:24.327702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:24.327715 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:24.327771 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:24.327959 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:24.327960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:24.327960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:24.328082 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:24.328205 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.328338 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.328412 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":66,"threads_started":1}
I20260929 03:29:24.331995 19386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:29:24.335798 19386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:29:24.343676 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.339249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:45417 (local address 127.0.0.1:58258)
0929 03:29:24.339800 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.339808 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.339849 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:24.340098 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:24.340099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:24.340101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:24.340131 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.340132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.341312 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.341313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.342342 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.342345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.343370 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.343371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.343447 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.343462 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 03:29:24.343524 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 03:29:24.343580 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":38,"thread_start_us":72,"threads_started":1}
I20260929 03:29:24.344079 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.339322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.6.116.131:45417)
0929 03:29:24.339507 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.339512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.339918 (+ 406us) server_negotiation.cc:407] Connection header received
0929 03:29:24.339950 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.339952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.339967 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.340030 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:24.340201 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.341229 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.342439 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.343218 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.343285 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.343338 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.343961 (+ 623us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.344005 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":62,"threads_started":1}
I20260929 03:29:24.344956 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.338740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45489 (local address 127.0.0.1:53842)
0929 03:29:24.339648 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.339657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.339718 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:24.339989 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:24.339992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:24.339994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:24.340051 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.340053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.341463 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.341464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.343408 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.343410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.344727 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.344728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.344771 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.344774 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:24.344856 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:29:24.344892 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":824,"thread_start_us":80,"threads_started":1}
I20260929 03:29:24.345000 19786 tablet_service.cc:1467] Processing CreateTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c (DEFAULT_TABLE table=test-table [id=3186136be1ab4a1c8fdc88f55d9af9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:24.344983 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.338989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.6.116.130:45489)
0929 03:29:24.339596 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.339600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.339701 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:29:24.339780 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.339783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.339800 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.339878 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:24.340126 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.341215 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.343493 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.344567 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.344634 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.344685 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.344874 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.344912 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":58,"threads_started":1}
I20260929 03:29:24.345397 19786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907b1fab51b4c448bf9e65d4542d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:24.345783 19652 tablet_service.cc:1467] Processing CreateTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c (DEFAULT_TABLE table=test-table [id=3186136be1ab4a1c8fdc88f55d9af9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:24.346108 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907b1fab51b4c448bf9e65d4542d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:24.349594 19885 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: Bootstrap starting.
I20260929 03:29:24.350199 19885 tablet_bootstrap.cc:654] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:24.350485 19885 log.cc:824] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:24.350797 19886 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Bootstrap starting.
I20260929 03:29:24.351735 19886 tablet_bootstrap.cc:654] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:24.351971 19886 log.cc:824] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:24.352680 19885 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: No bootstrap required, opened a new log
I20260929 03:29:24.352773 19885 ts_tablet_manager.cc:1397] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 03:29:24.353144 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.338740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:40671 (local address 127.0.0.1:34806)
0929 03:29:24.339173 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.339214 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.339275 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:24.349031 (+ 9756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:24.349032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:24.349034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:24.349071 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.349072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.350736 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.350737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.351801 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.351804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.352951 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.352952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.353009 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.353013 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:24.353045 (+ 32us) client_negotiation.cc:240] Negotiation successful
0929 03:29:24.353080 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":74,"threads_started":1}
I20260929 03:29:24.353603 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.338989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.6.116.129:40671)
0929 03:29:24.348846 (+ 9857us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.348850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.348862 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:29:24.348895 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.348897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.348912 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.349029 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:24.349164 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.350624 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.351964 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.352804 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.353416 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.353477 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.353501 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.353536 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9824,"thread_start_us":84,"threads_started":1}
I20260929 03:29:24.354341 19518 tablet_service.cc:1467] Processing CreateTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c (DEFAULT_TABLE table=test-table [id=3186136be1ab4a1c8fdc88f55d9af9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:24.354645 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0907b1fab51b4c448bf9e65d4542d21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:24.355237 19885 raft_consensus.cc:348] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.355408 19885 raft_consensus.cc:374] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:24.355445 19885 raft_consensus.cc:729] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d965e6ca2b243ce8573a02a3340572d, State: Initialized, Role: FOLLOWER
I20260929 03:29:24.355564 19885 consensus_queue.cc:260] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.355811 19885 ts_tablet_manager.cc:1428] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:29:24.355937 19852 heartbeater.cc:503] Master 127.6.116.190:39731 was elected leader, sending a full tablet report...
I20260929 03:29:24.357882 19890 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: Bootstrap starting.
I20260929 03:29:24.358461 19886 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: No bootstrap required, opened a new log
I20260929 03:29:24.358525 19886 ts_tablet_manager.cc:1397] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260929 03:29:24.358496 19890 tablet_bootstrap.cc:654] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:24.358878 19890 log.cc:824] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:24.360826 19886 raft_consensus.cc:348] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.360975 19886 raft_consensus.cc:374] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:24.361398 19886 raft_consensus.cc:729] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4378c938b8cd4a928ee92fc59ad8e2d9, State: Initialized, Role: FOLLOWER
I20260929 03:29:24.361521 19886 consensus_queue.cc:260] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.361765 19886 ts_tablet_manager.cc:1428] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:29:24.361826 19718 heartbeater.cc:503] Master 127.6.116.190:39731 was elected leader, sending a full tablet report...
I20260929 03:29:24.362340 19890 tablet_bootstrap.cc:492] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: No bootstrap required, opened a new log
I20260929 03:29:24.362401 19890 ts_tablet_manager.cc:1397] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260929 03:29:24.364492 19890 raft_consensus.cc:348] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.364665 19890 raft_consensus.cc:374] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:24.364706 19890 raft_consensus.cc:729] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5291f3066cad448fbec0fd6991b76ec0, State: Initialized, Role: FOLLOWER
I20260929 03:29:24.364806 19890 consensus_queue.cc:260] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.365031 19890 ts_tablet_manager.cc:1428] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:29:24.365068 19584 heartbeater.cc:503] Master 127.6.116.190:39731 was elected leader, sending a full tablet report...
W20260929 03:29:24.370183 19719 tablet.cc:2369] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:29:24.377817 19889 raft_consensus.cc:482] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:24.377907 19889 raft_consensus.cc:504] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.378201 19889 leader_election.cc:288] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5291f3066cad448fbec0fd6991b76ec0 (127.6.116.129:40671), 4378c938b8cd4a928ee92fc59ad8e2d9 (127.6.116.130:45489)
I20260929 03:29:24.381870 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.378519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:43309 (local address 127.6.116.129:40671)
0929 03:29:24.378595 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.378597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.378715 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:29:24.378757 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.378757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.378765 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.378816 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:24.379007 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.379700 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381121 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.381553 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381590 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.381596 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.381786 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.381817 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 03:29:24.382078 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.378354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:40671 (local address 127.6.116.131:43309)
0929 03:29:24.378619 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.378630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.378702 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:24.378858 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:24.378859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:24.378867 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:24.378925 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.378927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.379805 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.379805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.381036 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.381038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.381658 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.381722 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.381725 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:24.381965 (+ 240us) client_negotiation.cc:240] Negotiation successful
0929 03:29:24.382000 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":7,"thread_start_us":75,"threads_started":1}
I20260929 03:29:24.382311 19524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907b1fab51b4c448bf9e65d4542d21c" candidate_uuid: "7d965e6ca2b243ce8573a02a3340572d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5291f3066cad448fbec0fd6991b76ec0" is_pre_election: true
I20260929 03:29:24.382439 19524 raft_consensus.cc:2454] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d965e6ca2b243ce8573a02a3340572d in term 0.
I20260929 03:29:24.382465 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.378555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:60893 (local address 127.6.116.130:45489)
0929 03:29:24.378622 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.378624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.378838 (+ 214us) server_negotiation.cc:407] Connection header received
0929 03:29:24.378923 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.378924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.378932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.378958 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:24.379194 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.379924 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381235 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.381728 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381772 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.381779 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.382372 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.382412 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:29:24.382613 19740 leader_election.cc:302] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5291f3066cad448fbec0fd6991b76ec0, 7d965e6ca2b243ce8573a02a3340572d; no voters:
I20260929 03:29:24.382722 19889 raft_consensus.cc:2781] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:29:24.382761 19889 raft_consensus.cc:482] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:29:24.382786 19889 raft_consensus.cc:3037] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:24.382925 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.378364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45489 (local address 127.6.116.131:60893)
0929 03:29:24.378789 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.378797 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.378860 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:24.379029 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:24.379030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:24.379031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:24.379123 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.379124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.380022 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.380023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.381125 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:24.381128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.381846 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.381847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:24.382053 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.382057 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:24.382736 (+ 679us) client_negotiation.cc:240] Negotiation successful
0929 03:29:24.382853 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":96,"threads_started":1}
I20260929 03:29:24.383576 19889 raft_consensus.cc:504] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.383734 19889 leader_election.cc:288] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [CANDIDATE]: Term 1 election: Requested vote from peers 5291f3066cad448fbec0fd6991b76ec0 (127.6.116.129:40671), 4378c938b8cd4a928ee92fc59ad8e2d9 (127.6.116.130:45489)
I20260929 03:29:24.384023 19524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907b1fab51b4c448bf9e65d4542d21c" candidate_uuid: "7d965e6ca2b243ce8573a02a3340572d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5291f3066cad448fbec0fd6991b76ec0"
I20260929 03:29:24.384093 19524 raft_consensus.cc:3037] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:24.384093 19671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907b1fab51b4c448bf9e65d4542d21c" candidate_uuid: "7d965e6ca2b243ce8573a02a3340572d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9"
I20260929 03:29:24.384068 19672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0907b1fab51b4c448bf9e65d4542d21c" candidate_uuid: "7d965e6ca2b243ce8573a02a3340572d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" is_pre_election: true
I20260929 03:29:24.384168 19671 raft_consensus.cc:3037] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:24.384822 19524 raft_consensus.cc:2454] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d965e6ca2b243ce8573a02a3340572d in term 1.
I20260929 03:29:24.384951 19671 raft_consensus.cc:2454] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d965e6ca2b243ce8573a02a3340572d in term 1.
I20260929 03:29:24.384999 19740 leader_election.cc:302] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5291f3066cad448fbec0fd6991b76ec0, 7d965e6ca2b243ce8573a02a3340572d; no voters:
I20260929 03:29:24.385072 19889 raft_consensus.cc:2781] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:24.385192 19889 raft_consensus.cc:686] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 1 LEADER]: Becoming Leader. State: Replica: 7d965e6ca2b243ce8573a02a3340572d, State: Running, Role: LEADER
I20260929 03:29:24.385267 19889 consensus_queue.cc:237] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } }
I20260929 03:29:24.385985 19385 catalog_manager.cc:5495] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d965e6ca2b243ce8573a02a3340572d (127.6.116.131). New cstate: current_term: 1 leader_uuid: "7d965e6ca2b243ce8573a02a3340572d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d965e6ca2b243ce8573a02a3340572d" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 45417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20260929 03:29:24.402783 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.399842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58272 (local address 127.6.116.131:45417)
0929 03:29:24.399874 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:24.399875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:24.399881 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:29:24.399906 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:24.399906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:24.399920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:24.399968 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:29:24.400154 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.400807 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.401805 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:24.402088 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:24.402162 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:24.402319 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:29:24.402428 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:29:24.402491 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:24.402689 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 03:29:24.402726 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:29:24.406104 19671 raft_consensus.cc:1261] T 0907b1fab51b4c448bf9e65d4542d21c P 4378c938b8cd4a928ee92fc59ad8e2d9 [term 1 FOLLOWER]: Refusing update from remote peer 7d965e6ca2b243ce8573a02a3340572d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:24.406208 19524 raft_consensus.cc:1261] T 0907b1fab51b4c448bf9e65d4542d21c P 5291f3066cad448fbec0fd6991b76ec0 [term 1 FOLLOWER]: Refusing update from remote peer 7d965e6ca2b243ce8573a02a3340572d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:24.406392 19896 consensus_queue.cc:1035] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4378c938b8cd4a928ee92fc59ad8e2d9" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:24.406558 19889 consensus_queue.cc:1035] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5291f3066cad448fbec0fd6991b76ec0" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:24.410029 19899 mvcc.cc:205] Tried to move back new op lower bound from 7334512903805337600 to 7334512903722594304. Current Snapshot: MvccSnapshot[applied={T|T < 7334512903805337600}]
I20260929 03:29:24.410550 19900 mvcc.cc:205] Tried to move back new op lower bound from 7334512903805337600 to 7334512903722594304. Current Snapshot: MvccSnapshot[applied={T|T < 7334512903805337600}]
I20260929 03:29:24.413525 19385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:24.413610 19385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:24.414536 19385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f0e2cef9cd343f1b577631920ce95ee: Sending DeleteTablet for 3 replicas of tablet 0907b1fab51b4c448bf9e65d4542d21c
I20260929 03:29:24.414927 19786 tablet_service.cc:1514] Processing DeleteTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:24 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:58258
I20260929 03:29:24.415202 19518 tablet_service.cc:1514] Processing DeleteTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:24 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:34806
I20260929 03:29:24.415676 19652 tablet_service.cc:1514] Processing DeleteTablet for tablet 0907b1fab51b4c448bf9e65d4542d21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:24 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53842
I20260929 03:29:24.416111 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19424
I20260929 03:29:24.421312 19907 tablet_replica.cc:330] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d: stopping tablet replica
I20260929 03:29:24.421476 19907 raft_consensus.cc:2229] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:29:24.421604 19907 raft_consensus.cc:2258] T 0907b1fab51b4c448bf9e65d4542d21c P 7d965e6ca2b243ce8573a02a3340572d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 03:29:24.422573 19371 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:40671 (error 108)
W20260929 03:29:24.423133 19880 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:29:24.422971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:40671 (local address 127.0.0.1:34820)
0929 03:29:24.423027 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.423038 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:24.423090 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:40671: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:29:24.423236 19371 catalog_manager.cc:4584] TS 5291f3066cad448fbec0fd6991b76ec0 (127.6.116.129:40671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0907b1fab51b4c448bf9e65d4542d21c: Network error: Client connection negotiation failed: client connection to 127.6.116.129:40671: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260929 03:29:24.423492 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19588
I20260929 03:29:24.423506 19905 mvcc.cc:205] Tried to move back new op lower bound from 7334512903805337600 to 7334512903722594304. Current Snapshot: MvccSnapshot[applied={T|T < 7334512903805337600}]
I20260929 03:29:24.428722 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:24.428626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45489 (local address 127.0.0.1:53852)
0929 03:29:24.428661 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:24.428690 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:45489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 03:29:24.429544 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19721
I20260929 03:29:24.434751 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19353
I20260929 03:29:24.442646 6610 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 03:29:24.776765 6610 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 03:29:24.776841 6610 mini_ranger.cc:66] Stopping Ranger...
I20260929 03:29:25.121594 6610 mini_ranger.cc:68] Stopped Ranger
2026-09-29 03:29:25.121 UTC [18420] LOG: received smart shutdown request
2026-09-29 03:29:25.125 UTC [18420] LOG: background worker "logical replication launcher" (PID 18429) exited with exit code 1
2026-09-29 03:29:25.125 UTC [18424] LOG: shutting down
2026-09-29 03:29:25.267 UTC [18420] LOG: database system is shut down
2026-09-29T03:29:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45094 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:29:25 dist-test-slave-z2zg krb5kdc[19921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:29:25 dist-test-slave-z2zg krb5kdc[19921](info): set up 2 sockets
Sep 29 03:29:25 dist-test-slave-z2zg krb5kdc[19921](info): commencing operation
krb5kdc: starting...
W20260929 03:29:26.356834 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:29:26.461440 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43057
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:26.538172 19934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:26.538353 19934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:26.538376 19934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:26.539500 19934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:26.539541 19934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:26.539556 19934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:26.539569 19934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:26.539620 19934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:26.541043 19934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:26.541301 19934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:26.541527 19934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:26.546067 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:26.546243 19934 server_base.cc:1034] running on GCE node
W20260929 03:29:26.546099 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:26.546048 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:26.546595 19934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:26.546875 19934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:26.548040 19934 hybrid_clock.cc:648] HybridClock initialized: now 1790652566548011 us; error 51 us; skew 500 ppm
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:26.554373 19934 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:26.554847 19934 webserver.cc:460] Webserver started at http://127.6.116.190:38703/ using document root <none> and password file <none>
I20260929 03:29:26.555135 19934 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:26.555203 19934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:26.555335 19934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:26.556284 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "f552372e7d524260906ab5dae7f7e6b1"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "e9af31aa229ed685f7a8a90fe0758d3c"
server_key_iv: "18d448755d1043e33fb9ba56559ca50d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.556615 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "f552372e7d524260906ab5dae7f7e6b1"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "e9af31aa229ed685f7a8a90fe0758d3c"
server_key_iv: "18d448755d1043e33fb9ba56559ca50d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.557864 19934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:29:26.558543 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:26.558709 19934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:26.558810 19934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "f552372e7d524260906ab5dae7f7e6b1"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "e9af31aa229ed685f7a8a90fe0758d3c"
server_key_iv: "18d448755d1043e33fb9ba56559ca50d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.558892 19934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:26.572181 19934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:26.573657 19934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:26.573807 19934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:26.577761 19934 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:43057
I20260929 03:29:26.577791 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:43057 every 8 connection(s)
I20260929 03:29:26.578091 19934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:26.578830 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:26.580257 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:29:26.585556 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 19934
I20260929 03:29:26.585633 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:26.585829 6610 external_mini_cluster.cc:1428] Setting key c3851b8008b4fcafdd828325ca5fa716
I20260929 03:29:26.589252 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.580919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.6.116.190:43057)
0929 03:29:26.581056 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.581064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.581142 (+ 78us) server_negotiation.cc:407] Connection header received
0929 03:29:26.581624 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.581626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.581642 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.581715 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.582194 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.582964 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.583886 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.584181 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.584262 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.586230 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.586238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.586238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.586245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.587851 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.588286 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.588288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.588289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.588359 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.588591 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.588591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.588592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.588762 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.588818 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.589030 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.589113 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260929 03:29:26.589394 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.580876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:43057 (local address 127.0.0.1:36630)
0929 03:29:26.581020 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.581056 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.581504 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.581740 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.581742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.581958 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.582118 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.582121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.583126 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.583128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.583804 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.583806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.584257 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.584258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.584315 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.584729 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.584736 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.586144 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.587932 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.587939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.587941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.588209 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.588424 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.588425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.588425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.588532 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.588827 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.588841 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.588911 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.589002 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.589219 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":62,"thread_start_us":75,"threads_started":1}
I20260929 03:29:26.590245 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:29:26.590564 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.590479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60020)
0929 03:29:26.590510 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.590547 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 03:29:26.590725 19950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
I20260929 03:29:26.591142 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.591070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60026)
0929 03:29:26.591106 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.591128 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 03:29:26.591255 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:44405: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
I20260929 03:29:26.607820 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:44405
--webserver_interface=127.6.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:26.640472 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } attempt: 1
I20260929 03:29:26.640897 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.640775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60038)
0929 03:29:26.640834 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.640870 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:29:26.641202 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.641112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60044)
0929 03:29:26.641150 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.641173 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 03:29:26.641283 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:44405: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
W20260929 03:29:26.682591 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:26.682790 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:26.682827 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:26.683974 20009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:26.684034 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:26.684058 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:26.684090 20009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:26.684116 20009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:26.685568 20009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:26.685827 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:26.686053 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:26.690603 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:26.690618 20009 server_base.cc:1034] running on GCE node
W20260929 03:29:26.690580 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:26.690579 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:26.691095 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:26.691350 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:26.692495 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790652566692477 us; error 36 us; skew 500 ppm
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:26.694761 20009 init.cc:377] Logged in from keytab as kudu/127.6.116.189@KRBTEST.COM (short username kudu)
I20260929 03:29:26.695240 20009 webserver.cc:460] Webserver started at http://127.6.116.189:35631/ using document root <none> and password file <none>
I20260929 03:29:26.695420 20009 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:26.695459 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:26.695541 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:26.696467 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data/instance:
uuid: "803837c0d5484a3182ff48caed3c3c3b"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4c9222eedc7e16467b3b269c2d23dfe7"
server_key_iv: "195bdf4a5af1f66d32a73bca8cfb4c0b"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.696738 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal/instance:
uuid: "803837c0d5484a3182ff48caed3c3c3b"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4c9222eedc7e16467b3b269c2d23dfe7"
server_key_iv: "195bdf4a5af1f66d32a73bca8cfb4c0b"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.697880 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:29:26.698542 20024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:26.698745 20009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:26.698858 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "803837c0d5484a3182ff48caed3c3c3b"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4c9222eedc7e16467b3b269c2d23dfe7"
server_key_iv: "195bdf4a5af1f66d32a73bca8cfb4c0b"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.698930 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:26.709417 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } attempt: 2
I20260929 03:29:26.709789 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.709672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60054)
0929 03:29:26.709724 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.709765 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:29:26.710130 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.710023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60058)
0929 03:29:26.710064 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.710113 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 03:29:26.710209 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:44405: Network error: Client connection negotiation failed: client connection to 127.6.116.189:44405: connect: Connection refused (error 111)
I20260929 03:29:26.718354 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:26.720063 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:26.720213 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:26.724233 20009 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.189:44405
I20260929 03:29:26.724282 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.189:44405 every 8 connection(s)
I20260929 03:29:26.724610 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
I20260929 03:29:26.725358 20077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:26.726862 20077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:26.731973 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20009
I20260929 03:29:26.732050 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-1/wal/instance
I20260929 03:29:26.732220 6610 external_mini_cluster.cc:1428] Setting key 66b808c4f6543c6c51110cb60709f5cd
I20260929 03:29:26.734814 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.727576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.6.116.190:43057)
0929 03:29:26.727635 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.727637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.727831 (+ 194us) server_negotiation.cc:407] Connection header received
0929 03:29:26.728249 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.728250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.728261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.728313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.728816 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.729247 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.730071 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.730240 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.730287 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.732345 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.732346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.732347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.732354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.733530 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.733890 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.733891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.733892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.733955 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.734212 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.734213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.734213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.734321 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.734401 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.734651 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.734731 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:29:26.734978 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.727559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:43057 (local address 127.0.0.1:36646)
0929 03:29:26.727707 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.727745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.728173 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.728362 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.728364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.728574 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.728753 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.728757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.729317 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.729318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.730008 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.730009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.730305 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.730306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.730396 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.730779 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.730786 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.732267 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.733616 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.733623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.733625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.733820 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.734042 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.734043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.734043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.734147 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.734394 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.734400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.734529 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.734631 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.734769 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260929 03:29:26.735520 20077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.116.188@KRBTEST.COM; defaulting to no policy
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.188@KRBTEST.COM" created.
I20260929 03:29:26.742095 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.735756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60070)
0929 03:29:26.735801 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.735810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.735845 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.736079 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.736080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.736210 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.736246 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.736247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.737047 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.737049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.737516 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.737518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.737968 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.737969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.738030 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.738429 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.738434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.739551 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.741146 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.741146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.741147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.741288 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.741518 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.741519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.741519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.741576 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.741894 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.741895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.741927 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.741981 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.742041 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:29:26.742278 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.735834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60070 (local address 127.6.116.189:44405)
0929 03:29:26.735951 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.735954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.735968 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:29:26.735982 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.735983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.735993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.736079 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.736325 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.736965 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.737604 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.737887 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.737969 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.739645 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.739651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.739652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.739657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.741077 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.741354 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.741356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.741356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.741423 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.741638 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.741639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.741639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.741811 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.741909 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.742054 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.742138 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260929 03:29:26.742798 20077 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:29:26.743148 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.743008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37938)
0929 03:29:26.743071 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.743120 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 03:29:26.743279 20025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
I20260929 03:29:26.743670 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.743571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37950)
0929 03:29:26.743620 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.743648 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 03:29:26.743770 20077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:34703: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
I20260929 03:29:26.755499 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:34703
--webserver_interface=127.6.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:26.793004 20077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } attempt: 1
I20260929 03:29:26.793512 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.793370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37964)
0929 03:29:26.793435 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.793479 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:29:26.793889 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.793787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37978)
0929 03:29:26.793842 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.793869 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:29:26.794008 20077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:34703: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
I20260929 03:29:26.801342 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } attempt: 3
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:29:26.808214 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.801630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60078)
0929 03:29:26.801682 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.801693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.801745 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.801877 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.801878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.802029 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.802089 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.802090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.802842 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.802844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.803410 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.803412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.803801 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.803802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.803885 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.804319 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.804332 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.805501 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.807194 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.807195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.807196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.807373 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.807582 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.807583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.807583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.807651 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.807908 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.807909 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.807950 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.808033 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.808123 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:29:26.808212 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.801686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.6.116.189:44405)
0929 03:29:26.801740 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.801742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.801753 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:29:26.801795 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.801796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.801805 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.801871 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.802233 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.802751 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.803491 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.803718 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.803801 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.805587 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.805589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.805589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.805596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.807096 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.807459 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.807460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.807460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.807516 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.807720 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.807720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.807721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.807826 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.807901 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.808056 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.808125 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:29:26.808517 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:29:26.808832 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.808736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37990)
0929 03:29:26.808780 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.808808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:29:26.809158 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.809082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:37998)
0929 03:29:26.809125 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.809145 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:29:26.809257 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:34703: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
W20260929 03:29:26.834583 20084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:26.834751 20084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:26.834771 20084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:26.835943 20084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:26.835983 20084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:26.835996 20084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:26.836032 20084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:26.836057 20084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:26.837441 20084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43057,127.6.116.189:44405,127.6.116.188:34703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:26.837600 20084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:26.837812 20084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:29:26.840390 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } attempt: 1
I20260929 03:29:26.840754 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.840616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38010)
0929 03:29:26.840674 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.840722 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:29:26.841087 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.841005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38020)
0929 03:29:26.841034 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.841063 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260929 03:29:26.841210 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:34703: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
W20260929 03:29:26.842772 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:26.842757 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:26.842757 20092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:26.843071 20084 server_base.cc:1034] running on GCE node
I20260929 03:29:26.843256 20084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:26.843466 20084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:26.844633 20084 hybrid_clock.cc:648] HybridClock initialized: now 1790652566844623 us; error 25 us; skew 500 ppm
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:26.846892 20084 init.cc:377] Logged in from keytab as kudu/127.6.116.188@KRBTEST.COM (short username kudu)
I20260929 03:29:26.847358 20084 webserver.cc:460] Webserver started at http://127.6.116.188:44525/ using document root <none> and password file <none>
I20260929 03:29:26.847577 20084 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:26.847618 20084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:26.847755 20084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:26.848685 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data/instance:
uuid: "1434322e6a7242b1a407159abf54fefb"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4893fbed3f1577e7276ee8b6f8e3a061"
server_key_iv: "e92d81302c5a082d8a68e9803991098d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.849016 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal/instance:
uuid: "1434322e6a7242b1a407159abf54fefb"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4893fbed3f1577e7276ee8b6f8e3a061"
server_key_iv: "e92d81302c5a082d8a68e9803991098d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.850195 20084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:29:26.850812 20099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:26.850975 20084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:26.851083 20084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "1434322e6a7242b1a407159abf54fefb"
format_stamp: "Formatted at 2026-09-29 03:29:26 on dist-test-slave-z2zg"
server_key: "4893fbed3f1577e7276ee8b6f8e3a061"
server_key_iv: "e92d81302c5a082d8a68e9803991098d"
server_key_version: "encryptionkey@0"
I20260929 03:29:26.851166 20084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:26.862136 20077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } attempt: 2
I20260929 03:29:26.862485 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.862377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38028)
0929 03:29:26.862430 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.862461 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:29:26.862820 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.862731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38030)
0929 03:29:26.862774 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.862799 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:29:26.862950 20077 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:34703: Network error: Client connection negotiation failed: client connection to 127.6.116.188:34703: connect: Connection refused (error 111)
I20260929 03:29:26.864990 20084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:26.866612 20084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:26.866765 20084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:26.870471 20084 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.188:34703
I20260929 03:29:26.870497 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.188:34703 every 8 connection(s)
I20260929 03:29:26.870783 20084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
I20260929 03:29:26.871533 20152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:26.872761 20152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:26.879818 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20084
I20260929 03:29:26.879945 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790652420563207-6610-0/minicluster-data/master-2/wal/instance
I20260929 03:29:26.880122 6610 external_mini_cluster.cc:1428] Setting key 62b9d1c7153f5dcd0d44c29cd2c98a4b
I20260929 03:29:26.880900 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.873307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.6.116.190:43057)
0929 03:29:26.873364 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.873365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.873529 (+ 164us) server_negotiation.cc:407] Connection header received
0929 03:29:26.873927 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.873927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.873936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.873986 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.874516 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.874980 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.875926 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.876098 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.876149 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.878040 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.878041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.878042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.878047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.879458 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.879886 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.879887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.879888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.879979 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.880230 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.880231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.880231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.880350 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.880427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.880737 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.880817 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:29:26.881078 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.873276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:43057 (local address 127.0.0.1:36656)
0929 03:29:26.873422 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.873458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.873857 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.874047 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.874050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.874259 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.874433 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.874437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.875092 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.875093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.875859 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.875861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.876166 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.876167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.876256 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.876689 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.876697 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.877957 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.879536 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.879541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.879542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.879741 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.880060 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.880060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.880061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.880159 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.880419 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.880426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.880579 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.880700 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.880856 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260929 03:29:26.881644 20152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:29:26.888345 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.881875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:44405 (local address 127.0.0.1:60092)
0929 03:29:26.881919 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.881929 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.881964 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.882083 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.882084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.882206 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.882246 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.882247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.883029 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.883030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.883699 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.883701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.884198 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.884199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.884270 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.884594 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.884598 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.885691 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.887348 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.887349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.887349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.887538 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.887736 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.887737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.887737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.887791 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.888057 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.888057 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.888095 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.888178 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.888259 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:29:26.888356 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.881931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.6.116.189:44405)
0929 03:29:26.881987 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.881989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.881997 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:29:26.882021 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.882022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.882030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.882073 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.882316 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.882945 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.883798 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.884102 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.884197 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.885766 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.885767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.885767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.885773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.887268 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.887609 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.887610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.887610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.887662 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.887875 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.887876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.887877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.887982 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.888063 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.888201 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.888266 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:29:26.888933 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.880486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.6.116.188:34703)
0929 03:29:26.880633 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.880638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.880781 (+ 143us) server_negotiation.cc:407] Connection header received
0929 03:29:26.880841 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.880844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.880858 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.880929 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.881677 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.882441 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.883339 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.883752 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.883810 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.886447 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.886455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.886455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.886461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.887820 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.888167 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.888169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.888169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.888214 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.888417 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.888417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.888418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.888537 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.888681 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.888747 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.888819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260929 03:29:26.889040 20152 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:26.895618 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.889403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.6.116.188:34703)
0929 03:29:26.889433 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.889435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.889442 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:29:26.889496 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.889496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.889505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.889556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.889803 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.890440 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.891197 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.891462 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.891526 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.892978 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.892980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.892980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.892987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.894385 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.894763 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.894764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.894764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.894820 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.895087 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.895088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.895088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.895192 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.895241 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.895395 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.895494 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:29:26.895689 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.889373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38052)
0929 03:29:26.889407 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.889417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.889450 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.889557 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.889558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.889699 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.889733 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.889735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.890502 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.890502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.891139 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.891141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.891529 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.891530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.891561 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.891952 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.891956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.892903 (+ 947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.894474 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.894475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.894476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.894687 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.894892 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.894893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.894893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.894975 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.895258 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.895259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.895299 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.895380 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.895522 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":55}
I20260929 03:29:26.896793 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.889583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.6.116.190:43057)
0929 03:29:26.889614 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.889616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.889626 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:29:26.889679 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.889680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.889689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.889749 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.890278 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.890796 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.891614 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.891810 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.891895 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.893920 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.893921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.893922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.893928 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.895691 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.896045 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.896046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.896046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.896105 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.896297 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.896298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.896298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.896429 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.896593 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.896628 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.896696 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:29:26.897148 20152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: Bootstrap starting.
I20260929 03:29:26.898046 20152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:26.898337 20152 log.cc:824] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:26.898975 20152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: No bootstrap required, opened a new log
I20260929 03:29:26.900437 20152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.900564 20152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:26.900629 20152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1434322e6a7242b1a407159abf54fefb, State: Initialized, Role: FOLLOWER
I20260929 03:29:26.900779 20152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.901081 20152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:29:26.901047 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:26.902321 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:29:26.902777 20169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:29:26.902832 20169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:29:26.902963 20084 master_runner.cc:186] Error getting master registration for 127.6.116.190:43057: OK, instance_id { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" instance_seqno: 1790652566576708 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:29:26.904913 20084 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:29:26.905170 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.897657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.6.116.189:44405)
0929 03:29:26.897694 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.897696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.897702 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:29:26.897722 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.897723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.897730 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.897797 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.898258 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.898842 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.899537 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.899714 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.899805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.901983 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.901984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.901985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.901991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.903945 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.904334 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.904335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.904335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.904387 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.904610 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.904611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.904611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.904728 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.904990 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.905031 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.905099 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:29:26.906708 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:29:26.915207 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.907958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.6.116.189:44405)
0929 03:29:26.907988 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.907990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.908082 (+ 92us) server_negotiation.cc:407] Connection header received
0929 03:29:26.908207 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.908208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.908217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.908243 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.908935 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.909554 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.910465 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.910730 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.910765 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.912297 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.912299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.912299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.912306 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.913862 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.914294 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.914296 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.914296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.914349 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.914575 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.914576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.914576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.914691 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.914772 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.914993 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.915094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:29:26.915206 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.908239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.6.116.190:43057)
0929 03:29:26.908273 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.908275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.908446 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:29:26.908531 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.908532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.908540 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.908566 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.909201 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.909808 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.910755 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.911030 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.911093 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.912371 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.912372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.912373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.912379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.913972 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.914391 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.914393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.914393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.914444 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.914633 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.914634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.914634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.914741 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.914800 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.915002 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.915105 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:29:26.916273 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.908150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.6.116.188:34703)
0929 03:29:26.908184 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.908186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.908432 (+ 246us) server_negotiation.cc:407] Connection header received
0929 03:29:26.908517 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.908517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.908526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.908553 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.909104 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.909695 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.910537 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.910808 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.910843 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.913426 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.913427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.913427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.913433 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.915003 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.915442 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.915444 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.915444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.915500 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.915701 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.915701 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.915702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.915840 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.915933 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.916126 (+ 193us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.916191 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:29:26.916371 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } attempt: 2
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:29:26.922351 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.916567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38058)
0929 03:29:26.916616 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.916625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.916659 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.916770 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.916771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.916914 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.916946 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.916948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.917432 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.917432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.918120 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.918122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.918412 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.918413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.918462 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.918746 (+ 284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.918750 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.919984 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.921320 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.921320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.921321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.921522 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.921733 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.921734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.921734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.921795 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.922081 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.922081 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.922123 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.922191 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.922269 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:29:26.922359 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.916613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.6.116.188:34703)
0929 03:29:26.916641 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.916643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.916650 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:29:26.916702 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.916703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.916711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.916764 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.917006 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.917365 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.918195 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.918353 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.918393 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.920050 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.920050 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.920051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.920055 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.921253 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.921608 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.921609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.921610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.921661 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.921857 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.921858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.921858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.921965 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.922042 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.922214 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.922271 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:29:26.923525 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: Bootstrap starting.
I20260929 03:29:26.924022 20002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:26.924299 20002 log.cc:824] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:26.925058 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: No bootstrap required, opened a new log
I20260929 03:29:26.926460 20002 raft_consensus.cc:348] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.926558 20002 raft_consensus.cc:374] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:26.926577 20002 raft_consensus.cc:729] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f552372e7d524260906ab5dae7f7e6b1, State: Initialized, Role: FOLLOWER
I20260929 03:29:26.926703 20002 consensus_queue.cc:260] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.926976 20002 sys_catalog.cc:564] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:26.926962 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:26.927126 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:29:26.928597 20190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:29:26.928649 20190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:29:26.929171 19934 master_runner.cc:186] Error getting master registration for 127.6.116.189:44405: OK, instance_id { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" instance_seqno: 1790652566723102 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:29:26.931325 19934 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:29:26.954077 20077 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } attempt: 3
Sep 29 03:29:26 dist-test-slave-z2zg krb5kdc[19921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652566, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:29:26.960659 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.954334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:34703 (local address 127.0.0.1:38064)
0929 03:29:26.954375 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:26.954385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:26.954439 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:26.954589 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:26.954590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:26.954721 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:26.954764 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.954765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.955406 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.955407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.955873 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:26.955875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.956242 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.956243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:26.956295 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.956720 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:26.956725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:26.958068 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:26.959669 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.959670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.959670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.959841 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.960040 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:26.960041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:26.960041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:26.960096 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:26.960359 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:26.960360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:26.960401 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:29:26.960483 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:29:26.960571 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 03:29:26.960692 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:26.954396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.6.116.188:34703)
0929 03:29:26.954454 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:26.954456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:26.954464 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:29:26.954485 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:26.954486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:26.954495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:26.954572 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:26.954843 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.955323 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.955956 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:26.956176 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:26.956245 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:26.958161 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:26.958162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:26.958162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:26.958169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:26.959567 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.959922 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.959923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.959923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.959975 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:26.960179 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:26.960180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:26.960180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:26.960278 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:26.960363 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:26.960506 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 03:29:26.960571 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 03:29:26.962173 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: Bootstrap starting.
I20260929 03:29:26.962751 20077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:26.963075 20077 log.cc:824] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:26.963770 20077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: No bootstrap required, opened a new log
I20260929 03:29:26.965137 20077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.965260 20077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:26.965301 20077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 803837c0d5484a3182ff48caed3c3c3b, State: Initialized, Role: FOLLOWER
I20260929 03:29:26.965430 20077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:26.965688 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:26.965739 20077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:26.965811 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: This master's current role is: FOLLOWER
W20260929 03:29:26.967454 20204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:29:26.967547 20204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:29:26.970209 20009 master_runner.cc:263] No leader master found from master 803837c0d5484a3182ff48caed3c3c3b
I20260929 03:29:26.970252 20009 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 03:29:26.970798 20030 tablet.cc:2369] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:29:27.029883 20179 raft_consensus.cc:482] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:27.030002 20179 raft_consensus.cc:504] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:27.030208 20179 leader_election.cc:288] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 803837c0d5484a3182ff48caed3c3c3b (127.6.116.189:44405), 1434322e6a7242b1a407159abf54fefb (127.6.116.188:34703)
I20260929 03:29:27.030530 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f552372e7d524260906ab5dae7f7e6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1434322e6a7242b1a407159abf54fefb" is_pre_election: true
I20260929 03:29:27.030675 20126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f552372e7d524260906ab5dae7f7e6b1 in term 0.
I20260929 03:29:27.030964 19950 leader_election.cc:302] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1434322e6a7242b1a407159abf54fefb, f552372e7d524260906ab5dae7f7e6b1; no voters:
I20260929 03:29:27.031111 20179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:29:27.031155 20179 raft_consensus.cc:482] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:29:27.031176 20179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:27.032056 20179 raft_consensus.cc:504] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:27.032162 20179 leader_election.cc:288] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [CANDIDATE]: Term 1 election: Requested vote from peers 803837c0d5484a3182ff48caed3c3c3b (127.6.116.189:44405), 1434322e6a7242b1a407159abf54fefb (127.6.116.188:34703)
I20260929 03:29:27.032514 20051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f552372e7d524260906ab5dae7f7e6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803837c0d5484a3182ff48caed3c3c3b" is_pre_election: true
I20260929 03:29:27.032663 20051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f552372e7d524260906ab5dae7f7e6b1 in term 0.
I20260929 03:29:27.033046 20051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f552372e7d524260906ab5dae7f7e6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "803837c0d5484a3182ff48caed3c3c3b"
I20260929 03:29:27.033109 20051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:27.033907 20051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f552372e7d524260906ab5dae7f7e6b1 in term 1.
I20260929 03:29:27.034109 19950 leader_election.cc:302] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 803837c0d5484a3182ff48caed3c3c3b, f552372e7d524260906ab5dae7f7e6b1; no voters:
I20260929 03:29:27.034230 20126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f552372e7d524260906ab5dae7f7e6b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1434322e6a7242b1a407159abf54fefb"
I20260929 03:29:27.034297 20126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:27.035166 20179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:27.035336 20179 raft_consensus.cc:686] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [term 1 LEADER]: Becoming Leader. State: Replica: f552372e7d524260906ab5dae7f7e6b1, State: Running, Role: LEADER
I20260929 03:29:27.035422 20179 consensus_queue.cc:237] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } }
I20260929 03:29:27.035808 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f552372e7d524260906ab5dae7f7e6b1. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.035887 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:27.036911 20126 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f552372e7d524260906ab5dae7f7e6b1 in term 1.
I20260929 03:29:27.037814 20208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:27.038038 20208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:27.039476 20126 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [term 1 FOLLOWER]: Refusing update from remote peer f552372e7d524260906ab5dae7f7e6b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:27.039804 20206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:27.040942 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f552372e7d524260906ab5dae7f7e6b1. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.041016 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:29:27.041375 20051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [term 1 FOLLOWER]: Refusing update from remote peer f552372e7d524260906ab5dae7f7e6b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:27.041641 20206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:27.042680 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f552372e7d524260906ab5dae7f7e6b1. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.042752 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:29:27.043658 20206 sys_catalog.cc:455] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.043730 20206 sys_catalog.cc:458] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:27.044112 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.044179 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:29:27.044402 20209 mvcc.cc:205] Tried to move back new op lower bound from 7334512914591375360 to 7334512914577543168. Current Snapshot: MvccSnapshot[applied={T|T < 7334512914591375360}]
I20260929 03:29:27.044909 20212 mvcc.cc:205] Tried to move back new op lower bound from 7334512914591375360 to 7334512914577543168. Current Snapshot: MvccSnapshot[applied={T|T < 7334512914591375360}]
I20260929 03:29:27.045644 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.045715 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1 [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:27.046353 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f552372e7d524260906ab5dae7f7e6b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f552372e7d524260906ab5dae7f7e6b1" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43057 } } peers { permanent_uuid: "803837c0d5484a3182ff48caed3c3c3b" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 44405 } } peers { permanent_uuid: "1434322e6a7242b1a407159abf54fefb" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 34703 } } }
I20260929 03:29:27.046427 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:29:27.047119 20208 catalog_manager.cc:1347] Generated new cluster ID: 3285bd520a77459a82440fcc25585454
I20260929 03:29:27.047161 20208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:29:27.059525 20210 mvcc.cc:205] Tried to move back new op lower bound from 7334512914591375360 to 7334512914577543168. Current Snapshot: MvccSnapshot[applied={T|T < 7334512914591375360}]
I20260929 03:29:27.074812 20208 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:29:27.075564 20208 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:27.080751 20208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f552372e7d524260906ab5dae7f7e6b1: Generated new TSK 0
I20260929 03:29:27.080945 20208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:29:27.903231 20169 catalog_manager.cc:1259] Loaded cluster ID: 3285bd520a77459a82440fcc25585454
I20260929 03:29:27.903297 20169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: loading cluster ID for follower catalog manager: success
I20260929 03:29:27.904086 20169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: acquiring CA information for follower catalog manager: success
I20260929 03:29:27.904335 20169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1434322e6a7242b1a407159abf54fefb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:29:27.967975 20204 catalog_manager.cc:1259] Loaded cluster ID: 3285bd520a77459a82440fcc25585454
I20260929 03:29:27.968045 20204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: loading cluster ID for follower catalog manager: success
I20260929 03:29:27.968803 20204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: acquiring CA information for follower catalog manager: success
I20260929 03:29:27.969051 20204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 803837c0d5484a3182ff48caed3c3c3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:29:28.130160 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 19934
I20260929 03:29:28.135174 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20009
I20260929 03:29:28.139654 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20084
2026-09-29T03:29:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2766 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:29:28 dist-test-slave-z2zg krb5kdc[20224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:29:28 dist-test-slave-z2zg krb5kdc[20224](info): set up 2 sockets
Sep 29 03:29:28 dist-test-slave-z2zg krb5kdc[20224](info): commencing operation
krb5kdc: starting...
W20260929 03:29:29.132735 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:29:29.242089 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42967
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:42967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:29.317324 20237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:29.317541 20237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:29.317579 20237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:29.318666 20237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:29.318732 20237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:29.318755 20237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:29.318783 20237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:29.318812 20237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:29.320195 20237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:42967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:29.320452 20237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:29.320663 20237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:29.325066 20242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:29.325078 20245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:29.325078 20243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:29.325150 20237 server_base.cc:1034] running on GCE node
I20260929 03:29:29.325573 20237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:29.325835 20237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:29.326969 20237 hybrid_clock.cc:648] HybridClock initialized: now 1790652569326959 us; error 27 us; skew 500 ppm
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652569, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:29.329330 20237 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:29.329780 20237 webserver.cc:460] Webserver started at http://127.6.116.190:35645/ using document root <none> and password file <none>
I20260929 03:29:29.330005 20237 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:29.330047 20237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:29.330184 20237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:29.331198 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "2a848bc29de04e4aa956b7b9ff958925"
format_stamp: "Formatted at 2026-09-29 03:29:29 on dist-test-slave-z2zg"
server_key: "a9e78323fcc7e89e32fbb01803812c1c"
server_key_iv: "1785e18fb50b0c8453202b03eb72f4c3"
server_key_version: "encryptionkey@0"
I20260929 03:29:29.331529 20237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "2a848bc29de04e4aa956b7b9ff958925"
format_stamp: "Formatted at 2026-09-29 03:29:29 on dist-test-slave-z2zg"
server_key: "a9e78323fcc7e89e32fbb01803812c1c"
server_key_iv: "1785e18fb50b0c8453202b03eb72f4c3"
server_key_version: "encryptionkey@0"
I20260929 03:29:29.332729 20237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:29.333379 20252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:29.333539 20237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:29.333633 20237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "2a848bc29de04e4aa956b7b9ff958925"
format_stamp: "Formatted at 2026-09-29 03:29:29 on dist-test-slave-z2zg"
server_key: "a9e78323fcc7e89e32fbb01803812c1c"
server_key_iv: "1785e18fb50b0c8453202b03eb72f4c3"
server_key_version: "encryptionkey@0"
I20260929 03:29:29.333706 20237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:29.341252 20237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:29.343362 20237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:29.343505 20237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:29.347188 20237 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:42967
I20260929 03:29:29.347237 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:42967 every 8 connection(s)
I20260929 03:29:29.347548 20237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:29.348312 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:29.350227 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925: Bootstrap starting.
I20260929 03:29:29.350733 20305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:29.350952 20305 log.cc:824] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:29.351632 20305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925: No bootstrap required, opened a new log
I20260929 03:29:29.353024 20305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } }
I20260929 03:29:29.353139 20305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:29.353173 20305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a848bc29de04e4aa956b7b9ff958925, State: Initialized, Role: FOLLOWER
I20260929 03:29:29.353291 20305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } }
I20260929 03:29:29.353359 20305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:29.353379 20305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:29.353410 20305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:29.353837 20305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } }
I20260929 03:29:29.353915 20305 leader_election.cc:302] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a848bc29de04e4aa956b7b9ff958925; no voters:
I20260929 03:29:29.354064 20305 leader_election.cc:288] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:29:29.354087 20308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:29.354195 20308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [term 1 LEADER]: Becoming Leader. State: Replica: 2a848bc29de04e4aa956b7b9ff958925, State: Running, Role: LEADER
I20260929 03:29:29.354300 20305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:29.354332 20308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } }
I20260929 03:29:29.354764 20310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a848bc29de04e4aa956b7b9ff958925. Latest consensus state: current_term: 1 leader_uuid: "2a848bc29de04e4aa956b7b9ff958925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } } }
I20260929 03:29:29.354859 20310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:29.354835 20309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a848bc29de04e4aa956b7b9ff958925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a848bc29de04e4aa956b7b9ff958925" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42967 } } }
I20260929 03:29:29.354914 20309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925 [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:29.355315 20317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:29.355813 20317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:29.356119 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20237
I20260929 03:29:29.356208 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:29:29.356406 6610 external_mini_cluster.cc:1428] Setting key 83cda909d6edc2b418d19a3229ab0636
I20260929 03:29:29.359158 20317 catalog_manager.cc:1347] Generated new cluster ID: bc7403186ad94dde84728dd33dcb198b
I20260929 03:29:29.359210 20317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:29.363977 20317 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:29:29.364379 20317 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:29.371666 20317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a848bc29de04e4aa956b7b9ff958925: Generated new TSK 0
I20260929 03:29:29.371809 20317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:29:29.373020 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:29.356852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.6.116.190:42967)
0929 03:29:29.357013 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:29.357022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:29.357057 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:29:29.357185 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:29.357192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:29.357244 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:29.357428 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:29.358001 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:29.358848 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:29.359979 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:29.360593 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:29.360703 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:29.363195 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:29.363198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:29.363199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:29.363208 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:29.371593 (+ 8385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:29.371964 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:29.371967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:29.371967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:29.372050 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:29.372256 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:29.372256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:29.372257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:29.372444 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:29.372559 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:29.372675 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 03:29:29.372778 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260929 03:29:29.373872 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:29:29.376646 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20237
2026-09-29T03:29:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1235 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20332](info): set up 2 sockets
Sep 29 03:29:29 dist-test-slave-z2zg krb5kdc[20332](info): commencing operation
krb5kdc: starting...
W20260929 03:29:30.373164 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:29:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:29:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:29:30.479719 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:30.554848 20345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:30.555111 20345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:30.555166 20345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:30.556260 20345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:30.556303 20345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:30.556317 20345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:30.556330 20345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:30.556341 20345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:30.557680 20345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:30.557845 20345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:30.557996 20345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:29:30.562417 20345 server_base.cc:1034] running on GCE node
W20260929 03:29:30.562389 20351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:30.562386 20353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:30.562399 20350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:30.562906 20345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:30.563186 20345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:30.564347 20345 hybrid_clock.cc:648] HybridClock initialized: now 1790652570564329 us; error 29 us; skew 500 ppm
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:30.566665 20345 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:30.567135 20345 webserver.cc:460] Webserver started at http://127.6.116.190:38331/ using document root <none> and password file <none>
I20260929 03:29:30.567363 20345 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:30.567427 20345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:30.567553 20345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:30.568516 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "09f904b5c849468d8442c17be8c18b5b"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "7765cadba40e2ffa7c5d3a275101c002"
server_key_iv: "ed71822c06b0e560a33a9b1ff2c008dd"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.568853 20345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "09f904b5c849468d8442c17be8c18b5b"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "7765cadba40e2ffa7c5d3a275101c002"
server_key_iv: "ed71822c06b0e560a33a9b1ff2c008dd"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.570082 20345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:29:30.570712 20360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.570886 20345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:30.570974 20345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "09f904b5c849468d8442c17be8c18b5b"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "7765cadba40e2ffa7c5d3a275101c002"
server_key_iv: "ed71822c06b0e560a33a9b1ff2c008dd"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.571077 20345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:30.578821 20345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:30.580786 20345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:30.580952 20345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:30.584663 20345 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:41975
I20260929 03:29:30.584697 20412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:41975 every 8 connection(s)
I20260929 03:29:30.585016 20345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:30.585876 20413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:30.587808 20413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap starting.
I20260929 03:29:30.588299 20413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:30.588500 20413 log.cc:824] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:30.589058 20413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: No bootstrap required, opened a new log
I20260929 03:29:30.590420 20413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:30.590536 20413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:30.590556 20413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Initialized, Role: FOLLOWER
I20260929 03:29:30.590643 20413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:30.590719 20413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:30.590752 20413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:30.590799 20413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:30.591617 20413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:30.591728 20413 leader_election.cc:302] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f904b5c849468d8442c17be8c18b5b; no voters:
I20260929 03:29:30.591883 20413 leader_election.cc:288] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:29:30.591923 20416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:30.592072 20416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 LEADER]: Becoming Leader. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Running, Role: LEADER
I20260929 03:29:30.592123 20413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:30.592209 20416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:30.592581 20418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f904b5c849468d8442c17be8c18b5b. Latest consensus state: current_term: 1 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:30.592689 20418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:30.592841 20417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:30.593065 20417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:30.593176 20425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:30.593532 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20345
I20260929 03:29:30.593621 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:29:30.593678 20425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:30.593837 6610 external_mini_cluster.cc:1428] Setting key 5d4fe0f18e2405d05677100d7b2bea28
I20260929 03:29:30.595418 20425 catalog_manager.cc:1347] Generated new cluster ID: 2fe32fd8d6184b3188190c852be0fdad
I20260929 03:29:30.595465 20425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:30.608929 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:30.594237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.6.116.190:41975)
0929 03:29:30.594438 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:30.594447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:30.594549 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:29:30.594731 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:30.594738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:30.594817 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:30.594950 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:30.599475 (+ 4525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.601201 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.602229 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.602888 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.602993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:30.605598 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:30.605602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:30.605603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:30.605613 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:30.607451 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.607809 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.607812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.607812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.607897 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.608081 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.608082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.608082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.608326 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:30.608458 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:30.608553 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 03:29:30.608745 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260929 03:29:30.610890 20425 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:29:30.611511 20425 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:30.619390 20425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Generated new TSK 0
I20260929 03:29:30.619532 20425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:29:30.645814 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:30.725766 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:30.725966 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:30.725991 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:30.727111 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:30.727150 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:30.727201 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:30.728658 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:30.728900 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:30.729126 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:30.733973 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:30.734001 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:30.734163 20439 server_base.cc:1034] running on GCE node
W20260929 03:29:30.734184 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:30.734443 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:30.734680 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:30.735838 20439 hybrid_clock.cc:648] HybridClock initialized: now 1790652570735823 us; error 26 us; skew 500 ppm
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:30.738165 20439 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:30.738569 20439 webserver.cc:460] Webserver started at http://127.6.116.129:34993/ using document root <none> and password file <none>
I20260929 03:29:30.738790 20439 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:30.738859 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:30.738988 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:30.739974 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "83d6508750e54fce84b7deac483e3e52"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "69821867aa6692d9044911c418b7e478"
server_key_iv: "8eacb1ce71195fca5f555224a2896a7a"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.740342 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "83d6508750e54fce84b7deac483e3e52"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "69821867aa6692d9044911c418b7e478"
server_key_iv: "8eacb1ce71195fca5f555224a2896a7a"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.741570 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:29:30.742241 20454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.742414 20439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:30.742509 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "83d6508750e54fce84b7deac483e3e52"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "69821867aa6692d9044911c418b7e478"
server_key_iv: "8eacb1ce71195fca5f555224a2896a7a"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.742586 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:30.751406 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:30.753042 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:30.753196 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:30.753398 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:30.753674 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:30.753726 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.753764 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:30.753793 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.759425 20439 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:39979
I20260929 03:29:30.759491 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:39979 every 8 connection(s)
I20260929 03:29:30.759779 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:29:30.760727 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20439
I20260929 03:29:30.760835 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:29:30.761001 6610 external_mini_cluster.cc:1428] Setting key 43a8324d804cb8f32e633bee329dce52
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:29:30.769296 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:30.760462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:37787 (local address 127.6.116.190:41975)
0929 03:29:30.760524 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:30.760526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:30.760728 (+ 202us) server_negotiation.cc:407] Connection header received
0929 03:29:30.761307 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:30.761308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:30.761321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:30.761379 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:30.762323 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.762812 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.763950 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.764147 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.764198 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:30.766499 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:30.766501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:30.766501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:30.766508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:30.767912 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.768330 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.768331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.768332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.768390 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.768645 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.768646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.768647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.768755 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:30.768830 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:30.769124 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:29:30.769208 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:29:30.769449 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:30.760447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:37787)
0929 03:29:30.760622 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:30.760653 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:30.761216 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:30.761441 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:30.761445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:30.762062 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:30.762231 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:30.762235 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.762913 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.762914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:30.763859 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:30.763862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.764229 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.764230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:30.764385 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:30.764857 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:30.764867 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:30.766394 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:30.768009 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:30.768015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:30.768018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:30.768232 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:30.768459 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:30.768460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:30.768461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:30.768570 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:30.768828 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:30.768836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:30.768986 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 03:29:30.769094 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:29:30.769236 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260929 03:29:30.769937 20568 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:30.770009 20568 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:30.770184 20568 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:30.770731 20377 ts_manager.cc:194] Registered new tserver with Master: 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
I20260929 03:29:30.771522 20377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:37787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:29:30.785133 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:30.862473 20573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:30.862680 20573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:30.862715 20573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:30.863863 20573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:30.863925 20573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:30.863968 20573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:30.865360 20573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:30.865604 20573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:30.865819 20573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:30.870615 20579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:30.870738 20573 server_base.cc:1034] running on GCE node
W20260929 03:29:30.870662 20578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:30.870645 20581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:30.871116 20573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:30.871346 20573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:30.872483 20573 hybrid_clock.cc:648] HybridClock initialized: now 1790652570872470 us; error 27 us; skew 500 ppm
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:30.874818 20573 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:30.875226 20573 webserver.cc:460] Webserver started at http://127.6.116.130:45359/ using document root <none> and password file <none>
I20260929 03:29:30.875411 20573 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:30.875450 20573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:30.875543 20573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:30.876397 20573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "cf85b9b5753a9e0bfae8bc873b8b4c11"
server_key_iv: "8440e03aa8d3c623e515d407835b119e"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.876667 20573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "cf85b9b5753a9e0bfae8bc873b8b4c11"
server_key_iv: "8440e03aa8d3c623e515d407835b119e"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.877794 20573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:29:30.878387 20588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.878537 20573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:30.878625 20573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "cf85b9b5753a9e0bfae8bc873b8b4c11"
server_key_iv: "8440e03aa8d3c623e515d407835b119e"
server_key_version: "encryptionkey@0"
I20260929 03:29:30.878700 20573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:30.890919 20573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:30.892665 20573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:30.892793 20573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:30.893033 20573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:30.893373 20573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:30.893417 20573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.893436 20573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:30.893448 20573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:30.898780 20573 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:39095
I20260929 03:29:30.898849 20701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:39095 every 8 connection(s)
I20260929 03:29:30.899171 20573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:29:30.899504 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20573
I20260929 03:29:30.899595 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:29:30.899767 6610 external_mini_cluster.cc:1428] Setting key e5af939f5f10b421d0c296ad11a1663b
Sep 29 03:29:30 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:29:30.908635 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:30.899772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:52547 (local address 127.6.116.190:41975)
0929 03:29:30.899829 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:30.899831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:30.900039 (+ 208us) server_negotiation.cc:407] Connection header received
0929 03:29:30.900536 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:30.900537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:30.900545 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:30.900575 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:30.901724 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.902177 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.903427 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.903603 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.903679 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:30.905884 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:30.905886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:30.905886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:30.905893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:30.907322 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.907722 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.907723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.907724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.907779 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:30.908033 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:30.908034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:30.908034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:30.908152 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:30.908225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:30.908475 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 03:29:30.908550 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:29:30.908795 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:30.899763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:52547)
0929 03:29:30.899913 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:30.899949 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:30.900455 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:30.900651 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:30.900654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:30.901378 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:30.901640 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:30.901645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.902261 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.902263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:30.903358 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:30.903360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:30.903688 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:30.903689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:30.903806 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:30.904271 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:30.904278 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:30.905762 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:30.907424 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:30.907429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:30.907431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:30.907633 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:30.907857 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:30.907858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:30.907859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:30.907966 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:30.908216 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:30.908222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:30.908354 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 03:29:30.908455 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:29:30.908592 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260929 03:29:30.909243 20702 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:30.909309 20702 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:30.909492 20702 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:30.909842 20377 ts_manager.cc:194] Registered new tserver with Master: 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095)
I20260929 03:29:30.910216 20377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:52547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:29:30.923904 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:31.000566 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.000752 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.000775 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.001993 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.002038 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.002087 20707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:31.003506 20707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.003723 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.003930 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:31.008589 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.008603 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.008734 20707 server_base.cc:1034] running on GCE node
W20260929 03:29:31.008608 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.008989 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.009224 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.010372 20707 hybrid_clock.cc:648] HybridClock initialized: now 1790652571010358 us; error 44 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.012761 20707 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:31.013139 20707 webserver.cc:460] Webserver started at http://127.6.116.131:44133/ using document root <none> and password file <none>
I20260929 03:29:31.013348 20707 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.013414 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:31.013556 20707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:31.014487 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "b3b18419996e492ebe4ed291694af6d7"
format_stamp: "Formatted at 2026-09-29 03:29:31 on dist-test-slave-z2zg"
server_key: "6066557ed1b3e2f0e8e752864cb5fa7a"
server_key_iv: "3bfed4ccf136c6fa8529e077ef9ce593"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.014820 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "b3b18419996e492ebe4ed291694af6d7"
format_stamp: "Formatted at 2026-09-29 03:29:31 on dist-test-slave-z2zg"
server_key: "6066557ed1b3e2f0e8e752864cb5fa7a"
server_key_iv: "3bfed4ccf136c6fa8529e077ef9ce593"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.016093 20707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:29:31.016753 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.016928 20707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:31.017019 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "b3b18419996e492ebe4ed291694af6d7"
format_stamp: "Formatted at 2026-09-29 03:29:31 on dist-test-slave-z2zg"
server_key: "6066557ed1b3e2f0e8e752864cb5fa7a"
server_key_iv: "3bfed4ccf136c6fa8529e077ef9ce593"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.017098 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:31.027877 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:31.029824 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:31.029963 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:31.030150 20707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:31.030400 20707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:31.030448 20707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.030483 20707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:31.030506 20707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.036019 20707 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:39107
I20260929 03:29:31.036058 20835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:39107 every 8 connection(s)
I20260929 03:29:31.036312 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:29:31.037705 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20707
I20260929 03:29:31.037796 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:29:31.037995 6610 external_mini_cluster.cc:1428] Setting key 4a4c7f54fb99c8dac2cd78ac669fd050
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:31.044961 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.037004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:50583 (local address 127.6.116.190:41975)
0929 03:29:31.037055 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.037057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.037222 (+ 165us) server_negotiation.cc:407] Connection header received
0929 03:29:31.037671 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.037672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.037681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.037721 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.038859 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.039438 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.040329 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.040491 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.040539 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.042447 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.042448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.042448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.042453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.043888 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.044239 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.044240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.044240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.044286 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.044490 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.044491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.044491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.044580 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.044627 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.044842 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.044901 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:29:31.045087 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.036954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:50583)
0929 03:29:31.037101 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.037136 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.037579 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.037792 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.037796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.038518 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:31.038773 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.038778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.039527 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.039528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.040252 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.040254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.040577 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.040578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.040673 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.041050 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:31.041057 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:31.042368 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:31.043968 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.043973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.043974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.044175 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.044349 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.044350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.044350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.044437 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.044633 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:31.044637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:31.044733 (+ 96us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.044808 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.044914 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260929 03:29:31.045578 20836 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:31.045671 20836 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:31.045825 20836 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:31.046175 20377 ts_manager.cc:194] Registered new tserver with Master: b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
I20260929 03:29:31.046523 20377 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:50583
I20260929 03:29:31.049716 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:29:31.056524 20435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.050984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.6.116.190:41975)
0929 03:29:31.051083 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.051085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.051182 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:29:31.051252 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.051253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.051262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.051315 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.051849 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.052278 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.052969 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.053166 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.053237 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.054262 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.054264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.054264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.054271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.055391 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.055720 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.055721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.055721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.055779 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.055991 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.055992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.055992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.056130 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.056230 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.056357 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.056430 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:29:31.057934 20377 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:29:31.059500 20377 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:29:31.065567 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39107 (local address 127.0.0.1:34618)
0929 03:29:31.062015 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.062025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.062083 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.062239 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.062240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.062242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.062285 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.062287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063618 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.063619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.064488 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.064491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065268 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.065269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.065352 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.065366 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.065441 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.065497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":68,"threads_started":1}
I20260929 03:29:31.065577 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.6.116.131:39107)
0929 03:29:31.061912 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.061916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.062099 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:29:31.062136 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.062138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.062151 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.062227 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.062461 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063519 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.064574 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065180 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.065253 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.065300 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.065465 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.065503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260929 03:29:31.065856 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39979 (local address 127.0.0.1:42684)
0929 03:29:31.061870 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.061880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.061935 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.062123 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.062125 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.062127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.062188 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.062190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063574 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.063575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.064791 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.064793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065607 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.065608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.065681 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.065690 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.065748 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.065783 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":78,"thread_start_us":57,"threads_started":1}
I20260929 03:29:31.065951 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.6.116.129:39979)
0929 03:29:31.061804 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.061809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.061941 (+ 132us) server_negotiation.cc:407] Connection header received
0929 03:29:31.062005 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.062008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.062025 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.062102 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.062280 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063459 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.064899 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065468 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.065525 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.065574 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.065779 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.065836 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260929 03:29:31.065996 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.0.0.1:55392)
0929 03:29:31.061646 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.061686 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.061749 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.062348 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.062349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.062351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.062384 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.062385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063452 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.063453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.064799 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.064801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065691 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.065692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.065783 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.065787 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.065821 (+ 34us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.065859 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":61,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260929 03:29:31.066282 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.061836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55392 (local address 127.6.116.130:39095)
0929 03:29:31.062208 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.062213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.062222 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:29:31.062256 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.062258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.062272 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.062344 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.062482 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.063344 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.064883 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.065587 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.066061 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.066128 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.066149 (+ 21us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.066176 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":85,"threads_started":1}
I20260929 03:29:31.066560 20770 tablet_service.cc:1467] Processing CreateTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d (DEFAULT_TABLE table=test-table [id=b42bab26ac2340a28d81d564370489e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.066715 20502 tablet_service.cc:1467] Processing CreateTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d (DEFAULT_TABLE table=test-table [id=b42bab26ac2340a28d81d564370489e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.066884 20770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9119f3c2fe48b78a4ec82da2f0079d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.067005 20502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9119f3c2fe48b78a4ec82da2f0079d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.066962 20636 tablet_service.cc:1467] Processing CreateTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d (DEFAULT_TABLE table=test-table [id=b42bab26ac2340a28d81d564370489e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.067318 20636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db9119f3c2fe48b78a4ec82da2f0079d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.069641 20850 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Bootstrap starting.
I20260929 03:29:31.070034 20851 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Bootstrap starting.
I20260929 03:29:31.070184 20852 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Bootstrap starting.
I20260929 03:29:31.070329 20850 tablet_bootstrap.cc:654] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.070689 20851 tablet_bootstrap.cc:654] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.070798 20852 tablet_bootstrap.cc:654] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.071074 20850 log.cc:824] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.071373 20851 log.cc:824] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.071475 20852 log.cc:824] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.071904 20850 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: No bootstrap required, opened a new log
I20260929 03:29:31.072016 20850 ts_tablet_manager.cc:1397] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:29:31.072122 20851 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: No bootstrap required, opened a new log
I20260929 03:29:31.072184 20851 ts_tablet_manager.cc:1397] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:29:31.072261 20852 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: No bootstrap required, opened a new log
I20260929 03:29:31.072333 20852 ts_tablet_manager.cc:1397] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:29:31.074065 20850 raft_consensus.cc:348] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074044 20851 raft_consensus.cc:348] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074227 20851 raft_consensus.cc:374] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.074227 20850 raft_consensus.cc:374] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.074263 20851 raft_consensus.cc:729] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b18419996e492ebe4ed291694af6d7, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.074263 20850 raft_consensus.cc:729] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b88036e094df6b2d3727ae86cdd2d, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.074275 20852 raft_consensus.cc:348] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074383 20851 consensus_queue.cc:260] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074437 20852 raft_consensus.cc:374] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.074410 20850 consensus_queue.cc:260] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074463 20852 raft_consensus.cc:729] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d6508750e54fce84b7deac483e3e52, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.074565 20852 consensus_queue.cc:260] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.074651 20851 ts_tablet_manager.cc:1428] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:29:31.074651 20850 ts_tablet_manager.cc:1428] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:29:31.074759 20702 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
I20260929 03:29:31.074792 20836 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
I20260929 03:29:31.074824 20852 ts_tablet_manager.cc:1428] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:29:31.075014 20568 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
W20260929 03:29:31.149724 20703 tablet.cc:2369] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:29:31.172083 20858 raft_consensus.cc:482] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:31.172200 20858 raft_consensus.cc:504] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.172555 20858 leader_election.cc:288] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
I20260929 03:29:31.176419 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.172765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:49387 (local address 127.6.116.130:39095)
0929 03:29:31.172818 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.172821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.172831 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:29:31.172856 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.172857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.172874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.172946 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.173138 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.173911 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.175422 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.176012 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.176114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.176121 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.176322 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.176358 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:29:31.176419 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.172865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:60151 (local address 127.6.116.131:39107)
0929 03:29:31.172933 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.172935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.172944 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:29:31.172992 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.172993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.173001 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.173050 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.173212 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.173911 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.175422 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.176012 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.176114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.176121 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.176322 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.176360 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 03:29:31.176440 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.172684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.6.116.129:49387)
0929 03:29:31.172736 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.172749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.172793 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.172977 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.172978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.172980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.173075 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.173077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.173990 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.173990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.175334 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.175337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.176141 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.176142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.176257 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.176259 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.176306 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.176335 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":112}
I20260929 03:29:31.176448 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.172702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39107 (local address 127.6.116.129:60151)
0929 03:29:31.172867 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.172877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.172930 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.173070 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.173071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.173072 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.173131 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.173132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.173990 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.173990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.175334 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.175337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.176141 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.176142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.176231 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.176237 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.176296 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.176335 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":17,"thread_start_us":86,"threads_started":1}
I20260929 03:29:31.176792 20656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9119f3c2fe48b78a4ec82da2f0079d" candidate_uuid: "83d6508750e54fce84b7deac483e3e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" is_pre_election: true
I20260929 03:29:31.176908 20656 raft_consensus.cc:2454] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83d6508750e54fce84b7deac483e3e52 in term 0.
I20260929 03:29:31.176992 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9119f3c2fe48b78a4ec82da2f0079d" candidate_uuid: "83d6508750e54fce84b7deac483e3e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b18419996e492ebe4ed291694af6d7" is_pre_election: true
I20260929 03:29:31.177099 20456 leader_election.cc:302] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b88036e094df6b2d3727ae86cdd2d, 83d6508750e54fce84b7deac483e3e52; no voters:
I20260929 03:29:31.177145 20790 raft_consensus.cc:2454] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83d6508750e54fce84b7deac483e3e52 in term 0.
I20260929 03:29:31.177275 20858 raft_consensus.cc:2781] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:29:31.177330 20858 raft_consensus.cc:482] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:29:31.177346 20858 raft_consensus.cc:3037] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.178023 20858 raft_consensus.cc:504] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.178166 20858 leader_election.cc:288] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
I20260929 03:29:31.178304 20790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9119f3c2fe48b78a4ec82da2f0079d" candidate_uuid: "83d6508750e54fce84b7deac483e3e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b18419996e492ebe4ed291694af6d7"
I20260929 03:29:31.178318 20656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db9119f3c2fe48b78a4ec82da2f0079d" candidate_uuid: "83d6508750e54fce84b7deac483e3e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
I20260929 03:29:31.178375 20790 raft_consensus.cc:3037] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.178397 20656 raft_consensus.cc:3037] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.179109 20790 raft_consensus.cc:2454] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d6508750e54fce84b7deac483e3e52 in term 1.
I20260929 03:29:31.179210 20656 raft_consensus.cc:2454] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83d6508750e54fce84b7deac483e3e52 in term 1.
I20260929 03:29:31.179394 20457 leader_election.cc:302] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52, b3b18419996e492ebe4ed291694af6d7; no voters:
I20260929 03:29:31.179509 20858 raft_consensus.cc:2781] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:31.179678 20858 raft_consensus.cc:686] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 LEADER]: Becoming Leader. State: Replica: 83d6508750e54fce84b7deac483e3e52, State: Running, Role: LEADER
I20260929 03:29:31.179793 20858 consensus_queue.cc:237] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.180526 20376 catalog_manager.cc:5495] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83d6508750e54fce84b7deac483e3e52 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "83d6508750e54fce84b7deac483e3e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } health_report { overall_health: UNKNOWN } } }
I20260929 03:29:31.185060 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.182967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.6.116.129:39979)
0929 03:29:31.182998 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.183000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.183102 (+ 102us) server_negotiation.cc:407] Connection header received
0929 03:29:31.183174 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.183175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.183188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.183243 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:29:31.183327 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.183732 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.184403 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.184585 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.184635 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.184757 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:29:31.184854 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:29:31.184907 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.184942 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.184955 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:29:31.186527 20656 raft_consensus.cc:1261] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Refusing update from remote peer 83d6508750e54fce84b7deac483e3e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:31.186527 20790 raft_consensus.cc:1261] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Refusing update from remote peer 83d6508750e54fce84b7deac483e3e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:31.186795 20858 consensus_queue.cc:1035] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:31.186908 20860 consensus_queue.cc:1035] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:31.188936 20863 mvcc.cc:205] Tried to move back new op lower bound from 7334512931577843712 to 7334512931552919552. Current Snapshot: MvccSnapshot[applied={T|T < 7334512931577843712}]
I20260929 03:29:31.189277 20864 mvcc.cc:205] Tried to move back new op lower bound from 7334512931577843712 to 7334512931552919552. Current Snapshot: MvccSnapshot[applied={T|T < 7334512931577843712}]
I20260929 03:29:31.189740 20865 mvcc.cc:205] Tried to move back new op lower bound from 7334512931577843712 to 7334512931552919552. Current Snapshot: MvccSnapshot[applied={T|T < 7334512931577843712 or (T in {7334512931577843712})}]
I20260929 03:29:31.190945 20377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:31.191028 20377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:31.191779 20377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Sending DeleteTablet for 3 replicas of tablet db9119f3c2fe48b78a4ec82da2f0079d
I20260929 03:29:31.192083 20502 tablet_service.cc:1514] Processing DeleteTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:42684
I20260929 03:29:31.192101 20636 tablet_service.cc:1514] Processing DeleteTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:55392
I20260929 03:29:31.192184 20770 tablet_service.cc:1514] Processing DeleteTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:34618
I20260929 03:29:31.192231 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20439
I20260929 03:29:31.192759 20873 tablet_replica.cc:330] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: stopping tablet replica
I20260929 03:29:31.192836 20873 raft_consensus.cc:2229] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:31.192919 20873 raft_consensus.cc:2258] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:31.193055 20874 tablet_replica.cc:330] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: stopping tablet replica
I20260929 03:29:31.193130 20874 raft_consensus.cc:2229] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:31.193239 20874 raft_consensus.cc:2258] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:31.193382 20873 ts_tablet_manager.cc:1905] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:31.193611 20874 ts_tablet_manager.cc:1905] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:31.194672 20873 ts_tablet_manager.cc:1918] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:31.194751 20873 log.cc:1196] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/db9119f3c2fe48b78a4ec82da2f0079d
I20260929 03:29:31.195007 20873 ts_tablet_manager.cc:1939] T db9119f3c2fe48b78a4ec82da2f0079d P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting consensus metadata
I20260929 03:29:31.195434 20362 catalog_manager.cc:4841] TS 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): tablet db9119f3c2fe48b78a4ec82da2f0079d (table test-table [id=b42bab26ac2340a28d81d564370489e1]) successfully deleted
I20260929 03:29:31.195504 20874 ts_tablet_manager.cc:1918] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:31.195576 20874 log.cc:1196] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/db9119f3c2fe48b78a4ec82da2f0079d
I20260929 03:29:31.195847 20874 ts_tablet_manager.cc:1939] T db9119f3c2fe48b78a4ec82da2f0079d P b3b18419996e492ebe4ed291694af6d7: Deleting consensus metadata
I20260929 03:29:31.196244 20363 catalog_manager.cc:4841] TS b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): tablet db9119f3c2fe48b78a4ec82da2f0079d (table test-table [id=b42bab26ac2340a28d81d564370489e1]) successfully deleted
W20260929 03:29:31.198478 20363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:39979 (error 108)
I20260929 03:29:31.198697 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20573
I20260929 03:29:31.198863 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.198765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39979 (local address 127.0.0.1:42714)
0929 03:29:31.198805 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.198831 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:29:31.199023 20363 catalog_manager.cc:4584] TS 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db9119f3c2fe48b78a4ec82da2f0079d: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39979: connect: Connection refused (error 111)
I20260929 03:29:31.203765 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20707
I20260929 03:29:31.208194 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20345
I20260929 03:29:31.213263 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--webserver_interface=127.6.116.190
--webserver_port=38331
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:31.290747 20876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.290957 20876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.290995 20876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.292040 20876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:31.292119 20876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.292163 20876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.292225 20876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:31.292260 20876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:31.293637 20876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.293886 20876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.294097 20876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:31.302431 20885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.302438 20882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.302448 20883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.302474 20876 server_base.cc:1034] running on GCE node
I20260929 03:29:31.302918 20876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.303264 20876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.304430 20876 hybrid_clock.cc:648] HybridClock initialized: now 1790652571304418 us; error 44 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.310149 20876 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:31.310580 20876 webserver.cc:460] Webserver started at http://127.6.116.190:38331/ using document root <none> and password file <none>
I20260929 03:29:31.310763 20876 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.310819 20876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:31.312152 20876 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:31.312747 20892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.312919 20876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:31.313001 20876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "09f904b5c849468d8442c17be8c18b5b"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "7765cadba40e2ffa7c5d3a275101c002"
server_key_iv: "ed71822c06b0e560a33a9b1ff2c008dd"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.313381 20876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:31.333817 20876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:31.336050 20876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:31.336149 20876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:31.339865 20876 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:41975
I20260929 03:29:31.339910 20944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:41975 every 8 connection(s)
I20260929 03:29:31.340252 20876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:31.341727 20945 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:29:31.342340 20945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap starting.
I20260929 03:29:31.347688 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20876
I20260929 03:29:31.347954 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:39979
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=34993
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:31.350435 20945 log.cc:824] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.352975 20945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:29:31.353286 20945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap complete.
I20260929 03:29:31.356976 20945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:31.357365 20945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.357542 20945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:31.357628 20945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:31.357686 20945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:31.357758 20945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:29:31.359160 20945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:31.359261 20945 leader_election.cc:302] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f904b5c849468d8442c17be8c18b5b; no voters:
I20260929 03:29:31.359421 20945 leader_election.cc:288] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:29:31.359488 20949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:29:31.359639 20945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:31.359671 20949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 LEADER]: Becoming Leader. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Running, Role: LEADER
I20260929 03:29:31.359831 20949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:31.360126 20951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f904b5c849468d8442c17be8c18b5b. Latest consensus state: current_term: 2 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:31.361341 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:31.361416 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:31.360194 20951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:31.361912 20965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:31.362957 20965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:31.363091 20965 catalog_manager.cc:1259] Loaded cluster ID: 2fe32fd8d6184b3188190c852be0fdad
I20260929 03:29:31.363134 20965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:29:31.363426 20965 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 03:29:31.368971 20965 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 03:29:31.371583 20965 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:31.371760 20965 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Loaded TSK: 0
I20260929 03:29:31.372046 20965 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:29:31.458801 20947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.459092 20947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.459136 20947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.460685 20947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.460744 20947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.460783 20947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:31.462759 20947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=34993
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.462983 20947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.463263 20947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:31.468272 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.468295 20973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.468293 20976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.468547 20947 server_base.cc:1034] running on GCE node
I20260929 03:29:31.468740 20947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.468990 20947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.470177 20947 hybrid_clock.cc:648] HybridClock initialized: now 1790652571470163 us; error 26 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.473101 20947 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:31.473479 20947 webserver.cc:460] Webserver started at http://127.6.116.129:34993/ using document root <none> and password file <none>
I20260929 03:29:31.473716 20947 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.473794 20947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:31.475162 20947 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:31.475880 20983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.476061 20947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:31.476166 20947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "83d6508750e54fce84b7deac483e3e52"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "69821867aa6692d9044911c418b7e478"
server_key_iv: "8eacb1ce71195fca5f555224a2896a7a"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.476475 20947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:31.504925 20947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:31.507206 20947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:31.507352 20947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:31.507558 20947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:31.507977 20990 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:29:31.509660 20947 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:29:31.509738 20947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:29:31.509784 20947 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:29:31.510394 20947 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:29:31.510452 20947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:31.510502 20990 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Bootstrap starting.
I20260929 03:29:31.517993 20947 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:39979
I20260929 03:29:31.518096 21097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:39979 every 8 connection(s)
I20260929 03:29:31.518512 20947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:29:31.521601 20990 log.cc:824] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.523684 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 20947
I20260929 03:29:31.523767 20990 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:29:31.523938 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:39095
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=45359
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:31.524031 20990 tablet_bootstrap.cc:492] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Bootstrap complete.
I20260929 03:29:31.525228 20990 ts_tablet_manager.cc:1397] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260929 03:29:31.526566 20990 raft_consensus.cc:348] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
I20260929 03:29:31.526906 20990 raft_consensus.cc:729] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d6508750e54fce84b7deac483e3e52, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.527093 20990 consensus_queue.cc:260] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } }
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:31.527367 20990 ts_tablet_manager.cc:1428] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260929 03:29:31.531133 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.519263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:60135)
0929 03:29:31.519475 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.519523 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.520293 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.520808 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.520813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.521205 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:31.521452 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.521459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.522994 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.522995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.524254 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.524257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.525114 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.525115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.525261 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.525866 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:31.525877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:31.527672 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:31.529537 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.529544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.529546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.529769 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.530026 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.530027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.530027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.530135 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.530530 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:31.530537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:31.530662 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.530767 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.530925 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":115,"threads_started":1}
I20260929 03:29:31.531132 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.519567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:60135 (local address 127.6.116.190:41975)
0929 03:29:31.519790 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.519799 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.519835 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:29:31.520523 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.520531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.520583 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.520714 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.521571 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.522876 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.524350 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.525015 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.525625 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.527802 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.527805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.527806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.527817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.529428 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.529853 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.529857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.529857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.529949 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.530205 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.530206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.530207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.530439 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.530537 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.530831 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.530946 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260929 03:29:31.532004 21098 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:31.532097 21098 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:31.532300 21098 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:31.532979 20909 ts_manager.cc:194] Registered new tserver with Master: 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979)
I20260929 03:29:31.534065 20909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:60135
I20260929 03:29:31.535126 21098 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:29:31.541929 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.533638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39979 (local address 127.0.0.1:42730)
0929 03:29:31.533899 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.533936 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.534018 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.534255 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.534259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.534569 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:31.534606 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.534608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.535672 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.535673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.536676 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.536679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.537153 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.537154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.537243 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.537646 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:31.537651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:31.539113 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:31.540792 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.540798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.540800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.540992 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.541218 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.541219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.541219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.541282 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.541565 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:31.541571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:31.541618 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.541719 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.541830 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260929 03:29:31.541930 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.533623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.6.116.129:39979)
0929 03:29:31.533797 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.533801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.534021 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:29:31.534112 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.534114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.534129 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.534199 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.534687 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.535589 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.536780 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.537051 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.537108 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.539207 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.539214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.539214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.539220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.540687 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.541085 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.541088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.541088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.541145 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.541361 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.541362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.541362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.541492 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.541585 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.541753 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.541849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260929 03:29:31.542408 21032 tablet_service.cc:1514] Processing DeleteTablet for tablet db9119f3c2fe48b78a4ec82da2f0079d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:42730
I20260929 03:29:31.542855 21107 tablet_replica.cc:330] stopping tablet replica
I20260929 03:29:31.542953 21107 raft_consensus.cc:2229] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:31.543012 21107 raft_consensus.cc:2258] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:31.543316 21107 ts_tablet_manager.cc:1905] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:31.545411 21107 ts_tablet_manager.cc:1918] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:31.545492 21107 log.cc:1196] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/db9119f3c2fe48b78a4ec82da2f0079d
I20260929 03:29:31.545733 21107 ts_tablet_manager.cc:1939] T db9119f3c2fe48b78a4ec82da2f0079d P 83d6508750e54fce84b7deac483e3e52: Deleting consensus metadata
I20260929 03:29:31.546205 20895 catalog_manager.cc:4841] TS 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979): tablet db9119f3c2fe48b78a4ec82da2f0079d (table test-table [id=b42bab26ac2340a28d81d564370489e1]) successfully deleted
W20260929 03:29:31.610065 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.610245 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.610276 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.611428 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.611469 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.611495 21103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:31.612900 21103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:39095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=45359
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.613130 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.613353 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:31.618111 21115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.618134 21112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.618150 21103 server_base.cc:1034] running on GCE node
W20260929 03:29:31.618438 21113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.618661 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.618899 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.620056 21103 hybrid_clock.cc:648] HybridClock initialized: now 1790652571620041 us; error 31 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.625665 21103 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:31.626087 21103 webserver.cc:460] Webserver started at http://127.6.116.130:45359/ using document root <none> and password file <none>
I20260929 03:29:31.626331 21103 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.626412 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:31.627774 21103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:31.628324 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.628486 21103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:31.628587 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "cf85b9b5753a9e0bfae8bc873b8b4c11"
server_key_iv: "8440e03aa8d3c623e515d407835b119e"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.628901 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:31.648129 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:31.649739 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:31.649887 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:31.650110 21103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:31.650398 21103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:31.650449 21103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.650489 21103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:31.650517 21103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.656222 21103 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:39095
I20260929 03:29:31.656267 21235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:39095 every 8 connection(s)
I20260929 03:29:31.656590 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:31.663596 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21103
I20260929 03:29:31.663870 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:39107
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=44133
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:31.665263 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.657234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:48751 (local address 127.6.116.190:41975)
0929 03:29:31.657318 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.657321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.657485 (+ 164us) server_negotiation.cc:407] Connection header received
0929 03:29:31.657971 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.657972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.657984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.658036 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.658880 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.659443 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.660481 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.660699 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.660775 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.662613 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.662615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.662615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.662623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.663843 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.664288 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.664289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.664289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.664343 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.664587 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.664588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.664588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.664691 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.664764 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.665088 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.665196 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:29:31.665448 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.657202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:48751)
0929 03:29:31.657369 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.657412 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.657863 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.658097 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.658101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.658614 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:31.658785 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.658790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.659538 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.659539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.660365 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.660368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.660781 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.660783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.660916 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.661277 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:31.661284 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:31.662528 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:31.663928 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.663935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.663937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.664202 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.664414 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.664415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.664415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.664518 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.664764 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:31.664771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:31.664914 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.665035 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.665209 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260929 03:29:31.665935 21236 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:31.666054 21236 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:31.666229 21236 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:31.666600 20909 ts_manager.cc:194] Registered new tserver with Master: 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095)
I20260929 03:29:31.666988 20909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:48751
W20260929 03:29:31.740864 21239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.741083 21239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.741119 21239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.742177 21239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.742242 21239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.742302 21239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:31.743665 21239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:39107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=44133
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.743883 21239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.744089 21239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:29:31.748739 21239 server_base.cc:1034] running on GCE node
W20260929 03:29:31.748747 21244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.748697 21245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.748709 21247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.749089 21239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.749336 21239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.750479 21239 hybrid_clock.cc:648] HybridClock initialized: now 1790652571750467 us; error 28 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.752748 21239 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:31.753165 21239 webserver.cc:460] Webserver started at http://127.6.116.131:44133/ using document root <none> and password file <none>
I20260929 03:29:31.753387 21239 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.753464 21239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:31.754634 21239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:29:31.755215 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.755373 21239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:31.755463 21239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "b3b18419996e492ebe4ed291694af6d7"
format_stamp: "Formatted at 2026-09-29 03:29:31 on dist-test-slave-z2zg"
server_key: "6066557ed1b3e2f0e8e752864cb5fa7a"
server_key_iv: "3bfed4ccf136c6fa8529e077ef9ce593"
server_key_version: "encryptionkey@0"
I20260929 03:29:31.755757 21239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:31.775614 21239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:31.777410 21239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:31.777580 21239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:31.777794 21239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:31.778092 21239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:31.778144 21239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.778184 21239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:31.778211 21239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:31.783694 21239 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:39107
I20260929 03:29:31.783742 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:39107 every 8 connection(s)
I20260929 03:29:31.784031 21239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:29:31.788998 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21239
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:31.793202 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.784706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:59303 (local address 127.6.116.190:41975)
0929 03:29:31.784770 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.784772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.784975 (+ 203us) server_negotiation.cc:407] Connection header received
0929 03:29:31.785402 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.785403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.785413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.785462 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.786338 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.786829 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.787824 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.788025 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.788080 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.790301 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.790302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.790303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.790308 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.791743 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.792157 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.792157 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.792158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.792216 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.792562 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.792563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.792563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.792697 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.792764 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.793084 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.793143 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:29:31.793466 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.784666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:59303)
0929 03:29:31.784850 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.784889 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.785324 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.785515 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.785518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.786057 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:31.786228 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.786233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.786912 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.786913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.787743 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.787745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.788097 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.788098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.788191 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.788621 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:31.788628 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:31.790221 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:31.791819 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.791825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.791828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.792099 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.792333 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:31.792333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:31.792334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:31.792468 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:31.792767 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:31.792772 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:31.792929 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.793050 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.793215 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260929 03:29:31.793926 21368 heartbeater.cc:347] Connected to a master server at 127.6.116.190:41975
I20260929 03:29:31.794018 21368 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:31.794189 21368 heartbeater.cc:511] Master 127.6.116.190:41975 requested a full tablet report, sending...
I20260929 03:29:31.794646 20909 ts_manager.cc:194] Registered new tserver with Master: b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
I20260929 03:29:31.795178 20909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:59303
I20260929 03:29:31.796355 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.789416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.6.116.190:41975)
0929 03:29:31.789500 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.789502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.789599 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:29:31.789658 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.789659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.789667 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.789719 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:31.790382 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.791139 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.792006 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.792240 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.792318 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.793672 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:31.793673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:31.793674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:31.793680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:31.795319 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.795694 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.795694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.795695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.795743 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:31.795938 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:31.795938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:31.795938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:31.796054 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:31.796157 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.796229 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.796284 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":43,"threads_started":1}
I20260929 03:29:31.796968 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:29:31.799391 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.797306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.6.116.190:41975)
0929 03:29:31.797338 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.797340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.797388 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:29:31.797445 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.797445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.797469 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.797524 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:29:31.797588 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.797994 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.798690 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.798879 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.798944 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.799141 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:29:31.799221 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:29:31.799285 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.799331 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.799350 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:29:31.799651 20909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:29:31.801167 20909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:29:31.803388 21032 tablet_service.cc:1467] Processing CreateTablet for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f (DEFAULT_TABLE table=test-table [id=750d0f8dddc54e5da013017a6d4e67f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.803586 21032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.805099 20990 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Bootstrap starting.
I20260929 03:29:31.805402 20990 tablet_bootstrap.cc:654] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.805891 20990 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: No bootstrap required, opened a new log
I20260929 03:29:31.805949 20990 ts_tablet_manager.cc:1397] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:31.806298 20990 raft_consensus.cc:348] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.806372 20990 raft_consensus.cc:374] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.806398 20990 raft_consensus.cc:729] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d6508750e54fce84b7deac483e3e52, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.806458 20990 consensus_queue.cc:260] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.806546 20990 ts_tablet_manager.cc:1428] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:31.806934 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.802963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39107 (local address 127.0.0.1:34620)
0929 03:29:31.803299 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.803318 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.803374 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.803580 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.803581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.803583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.803629 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.803630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.804678 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.804679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.805823 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.805825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.806697 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.806698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.806758 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.806777 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.806826 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.806866 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":6,"thread_start_us":132,"threads_started":1}
I20260929 03:29:31.807188 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.803172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.6.116.131:39107)
0929 03:29:31.803309 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.803313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.803361 (+ 48us) server_negotiation.cc:407] Connection header received
0929 03:29:31.803434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.803436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.803447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.803701 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.803759 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.804579 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.806056 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.806628 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.806696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.806757 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.807068 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.807118 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260929 03:29:31.807777 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.803030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.6.116.130:39095)
0929 03:29:31.803505 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.803509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.803518 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:29:31.803555 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.803557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.803573 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.803653 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.803743 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.804819 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.806657 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.807457 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.807529 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.807564 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.807678 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.807705 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":102,"threads_started":1}
I20260929 03:29:31.807796 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.802922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.0.0.1:55394)
0929 03:29:31.802951 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.802961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.802997 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.803647 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.803648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.803650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.803682 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.803684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.804926 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.804927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.805976 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.805978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.807534 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.807535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.807603 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.807606 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.807670 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.807708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 03:29:31.807876 21302 tablet_service.cc:1467] Processing CreateTablet for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f (DEFAULT_TABLE table=test-table [id=750d0f8dddc54e5da013017a6d4e67f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.808187 21302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.808457 21170 tablet_service.cc:1467] Processing CreateTablet for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f (DEFAULT_TABLE table=test-table [id=750d0f8dddc54e5da013017a6d4e67f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:31.808739 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:31.811625 21378 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Bootstrap starting.
I20260929 03:29:31.812060 21379 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Bootstrap starting.
I20260929 03:29:31.812225 21378 tablet_bootstrap.cc:654] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.812636 21379 tablet_bootstrap.cc:654] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:31.812677 21378 log.cc:824] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.813093 21379 log.cc:824] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:31.813323 21378 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: No bootstrap required, opened a new log
I20260929 03:29:31.813397 21378 ts_tablet_manager.cc:1397] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:29:31.813773 21379 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: No bootstrap required, opened a new log
I20260929 03:29:31.813850 21379 ts_tablet_manager.cc:1397] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:29:31.814980 21378 raft_consensus.cc:348] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.815166 21378 raft_consensus.cc:374] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.815193 21378 raft_consensus.cc:729] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b18419996e492ebe4ed291694af6d7, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.815326 21378 consensus_queue.cc:260] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.815616 21378 ts_tablet_manager.cc:1428] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:29:31.815639 21368 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
I20260929 03:29:31.815536 21379 raft_consensus.cc:348] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.815698 21379 raft_consensus.cc:374] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:31.815732 21379 raft_consensus.cc:729] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b5b88036e094df6b2d3727ae86cdd2d, State: Initialized, Role: FOLLOWER
I20260929 03:29:31.815887 21379 consensus_queue.cc:260] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.816177 21379 ts_tablet_manager.cc:1428] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 03:29:31.816203 21236 heartbeater.cc:503] Master 127.6.116.190:41975 was elected leader, sending a full tablet report...
I20260929 03:29:31.857682 21382 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:31.857805 21382 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.858254 21382 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979)
I20260929 03:29:31.861712 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.858422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.6.116.131:39593)
0929 03:29:31.858480 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.858492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.858537 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.858705 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.858706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.858708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.858789 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.858791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.859607 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.859609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.860697 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.860700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.861447 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.861448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.861531 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.861534 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.861599 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.861640 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":10}
I20260929 03:29:31.861712 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.858502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:39593 (local address 127.6.116.130:39095)
0929 03:29:31.858549 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.858552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.858564 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:29:31.858595 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.858596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.858607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.858678 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.858891 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.859547 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.860780 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.861360 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.861449 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.861458 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.861617 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.861653 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:29:31.862025 21190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ccb60bb6e5488e9fa3d2aaa7f3516f" candidate_uuid: "b3b18419996e492ebe4ed291694af6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" is_pre_election: true
I20260929 03:29:31.862097 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.858525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:38797 (local address 127.6.116.129:39979)
0929 03:29:31.858574 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.858576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.858709 (+ 133us) server_negotiation.cc:407] Connection header received
0929 03:29:31.858818 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.858819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.858849 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.858920 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:31.859098 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.859808 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.861075 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.861619 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.861704 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.861782 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.861989 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.862037 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:29:31.862105 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.858455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39979 (local address 127.6.116.131:38797)
0929 03:29:31.858631 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.858661 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:31.858735 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:31.858944 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:31.858945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:31.858946 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:31.859005 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.859007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.859907 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.859908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.860957 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:31.860960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.861827 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.861828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:31.861915 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.861919 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:31.861985 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 03:29:31.862028 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260929 03:29:31.862171 21190 raft_consensus.cc:2454] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b18419996e492ebe4ed291694af6d7 in term 0.
I20260929 03:29:31.862419 21256 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7; no voters:
I20260929 03:29:31.862481 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ccb60bb6e5488e9fa3d2aaa7f3516f" candidate_uuid: "b3b18419996e492ebe4ed291694af6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d6508750e54fce84b7deac483e3e52" is_pre_election: true
I20260929 03:29:31.862540 21382 raft_consensus.cc:2781] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:29:31.862598 21382 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:29:31.862629 21382 raft_consensus.cc:3037] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.862639 21052 raft_consensus.cc:2454] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b18419996e492ebe4ed291694af6d7 in term 0.
I20260929 03:29:31.863464 21382 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.863619 21382 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [CANDIDATE]: Term 1 election: Requested vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979)
I20260929 03:29:31.863780 21190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ccb60bb6e5488e9fa3d2aaa7f3516f" candidate_uuid: "b3b18419996e492ebe4ed291694af6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
I20260929 03:29:31.863801 21052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48ccb60bb6e5488e9fa3d2aaa7f3516f" candidate_uuid: "b3b18419996e492ebe4ed291694af6d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d6508750e54fce84b7deac483e3e52"
I20260929 03:29:31.863865 21190 raft_consensus.cc:3037] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.863909 21052 raft_consensus.cc:3037] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:31.864502 21190 raft_consensus.cc:2454] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b18419996e492ebe4ed291694af6d7 in term 1.
I20260929 03:29:31.864639 21052 raft_consensus.cc:2454] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b18419996e492ebe4ed291694af6d7 in term 1.
I20260929 03:29:31.864648 21256 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7; no voters:
I20260929 03:29:31.864858 21382 raft_consensus.cc:2781] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:31.865010 21382 raft_consensus.cc:686] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 1 LEADER]: Becoming Leader. State: Replica: b3b18419996e492ebe4ed291694af6d7, State: Running, Role: LEADER
I20260929 03:29:31.865115 21382 consensus_queue.cc:237] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:31.865963 20908 catalog_manager.cc:5495] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b18419996e492ebe4ed291694af6d7 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "b3b18419996e492ebe4ed291694af6d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } health_report { overall_health: UNKNOWN } } }
I20260929 03:29:31.879398 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.877235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.6.116.131:39107)
0929 03:29:31.877295 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:31.877296 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:31.877311 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:29:31.877336 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:31.877337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:31.877350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:31.877407 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:29:31.877523 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.877944 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.878602 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:31.878790 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:31.878842 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:31.878980 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:29:31.879130 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:29:31.879193 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:31.879265 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:29:31.879284 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:29:31.880990 21052 raft_consensus.cc:1261] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Refusing update from remote peer b3b18419996e492ebe4ed291694af6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:31.881012 21190 raft_consensus.cc:1261] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Refusing update from remote peer b3b18419996e492ebe4ed291694af6d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:31.881273 21382 consensus_queue.cc:1035] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:31.881438 21385 consensus_queue.cc:1035] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:31.883724 21389 mvcc.cc:205] Tried to move back new op lower bound from 7334512934422224896 to 7334512934360330240. Current Snapshot: MvccSnapshot[applied={T|T < 7334512934422224896}]
I20260929 03:29:31.885565 20908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:31.885649 20908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:31.886390 20908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Sending DeleteTablet for 3 replicas of tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:31.886691 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20947
I20260929 03:29:31.886686 21170 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:55394
I20260929 03:29:31.889021 21302 tablet_service.cc:1514] Processing DeleteTablet for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:31 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:34620
I20260929 03:29:31.891160 21398 tablet_replica.cc:330] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: stopping tablet replica
I20260929 03:29:31.891161 21397 tablet_replica.cc:330] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: stopping tablet replica
I20260929 03:29:31.891268 21397 raft_consensus.cc:2229] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:31.891359 21398 raft_consensus.cc:2229] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:29:31.891372 21397 raft_consensus.cc:2258] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:31.891594 21398 raft_consensus.cc:2258] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:31.891775 21397 ts_tablet_manager.cc:1905] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:31.892113 21398 ts_tablet_manager.cc:1905] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:29:31.892155 20895 connection.cc:535] client connection to 127.6.116.129:39979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 03:29:31.892246 20895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:29:31.892845 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21103
I20260929 03:29:31.895398 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.895278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39979 (local address 127.0.0.1:42744)
0929 03:29:31.895329 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.895363 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:29:31.895560 20895 catalog_manager.cc:4584] TS 83d6508750e54fce84b7deac483e3e52 (127.6.116.129:39979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48ccb60bb6e5488e9fa3d2aaa7f3516f: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39979: connect: Connection refused (error 111)
I20260929 03:29:31.898838 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21239
I20260929 03:29:31.899022 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.898899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.0.0.1:55402)
0929 03:29:31.898943 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.898990 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:29:31.903331 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 20876
I20260929 03:29:31.903376 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:31.903273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39107 (local address 127.0.0.1:34640)
0929 03:29:31.903323 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:31.903356 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.131:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:29:31.908174 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--webserver_interface=127.6.116.190
--webserver_port=38331
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:31.985993 21400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:31.986176 21400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:31.986196 21400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:31.987450 21400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:31.987491 21400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:31.987506 21400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:31.987519 21400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:31.987531 21400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:31.988972 21400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=38331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:31.989207 21400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:31.989465 21400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:31.994122 21408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.994114 21406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:31.994098 21405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:31.994400 21400 server_base.cc:1034] running on GCE node
I20260929 03:29:31.994613 21400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:31.994915 21400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:31.996069 21400 hybrid_clock.cc:648] HybridClock initialized: now 1790652571996061 us; error 39 us; skew 500 ppm
Sep 29 03:29:31 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:31.998253 21400 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:31.998723 21400 webserver.cc:460] Webserver started at http://127.6.116.190:38331/ using document root <none> and password file <none>
I20260929 03:29:31.998946 21400 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:31.999013 21400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:32.000329 21400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:29:32.000998 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.001159 21400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:32.001257 21400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "09f904b5c849468d8442c17be8c18b5b"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "7765cadba40e2ffa7c5d3a275101c002"
server_key_iv: "ed71822c06b0e560a33a9b1ff2c008dd"
server_key_version: "encryptionkey@0"
I20260929 03:29:32.001562 21400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:32.018007 21400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:32.019721 21400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:32.019870 21400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:32.023746 21400 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:41975
I20260929 03:29:32.023788 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:41975 every 8 connection(s)
I20260929 03:29:32.024135 21400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:32.025555 21468 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:29:32.026191 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap starting.
I20260929 03:29:32.032644 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21400
I20260929 03:29:32.032981 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:39979
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=34993
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:32.034727 21468 log.cc:824] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:32.036865 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:29:32.037132 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b: Bootstrap complete.
I20260929 03:29:32.039227 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:32.039446 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Initialized, Role: FOLLOWER
I20260929 03:29:32.039592 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:32.039644 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:32.039696 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:32.039736 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:29:32.040650 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:32.040726 21468 leader_election.cc:302] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f904b5c849468d8442c17be8c18b5b; no voters:
I20260929 03:29:32.040941 21468 leader_election.cc:288] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:29:32.040972 21472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:29:32.041077 21472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [term 3 LEADER]: Becoming Leader. State: Replica: 09f904b5c849468d8442c17be8c18b5b, State: Running, Role: LEADER
I20260929 03:29:32.041165 21472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } }
I20260929 03:29:32.041190 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:32.041584 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f904b5c849468d8442c17be8c18b5b. Latest consensus state: current_term: 3 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:32.041657 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:32.041903 21476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:32.042331 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "09f904b5c849468d8442c17be8c18b5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f904b5c849468d8442c17be8c18b5b" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41975 } } }
I20260929 03:29:32.042418 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f904b5c849468d8442c17be8c18b5b [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:32.042678 21476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:32.042778 21476 catalog_manager.cc:1259] Loaded cluster ID: 2fe32fd8d6184b3188190c852be0fdad
I20260929 03:29:32.042815 21476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 03:29:32.043241 21476 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f114cfc19ad google::LogMessage::Fail() at ??:0
@ 0x7f114cfc488a google::LogMessage::SendToLog() at ??:0
@ 0x7f114cfc14de google::LogMessage::Flush() at ??:0
@ 0x7f114cfc2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1154ee60f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1154eef8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f114ee8f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f114ee84954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f114fce86db start_thread at ??:0
@ 0x7f114babf61f clone at ??:0
W20260929 03:29:32.127529 21470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:32.127717 21470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:32.127743 21470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:32.128882 21470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:32.128924 21470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:32.128952 21470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:32.130383 21470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:39979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=34993
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:32.130613 21470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:32.130848 21470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:32.135586 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.135618 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.135841 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:32.135953 21470 server_base.cc:1034] running on GCE node
I20260929 03:29:32.136163 21470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:32.136413 21470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:32.137567 21470 hybrid_clock.cc:648] HybridClock initialized: now 1790652572137556 us; error 27 us; skew 500 ppm
Sep 29 03:29:32 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:32.140225 21470 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:32.140640 21470 webserver.cc:460] Webserver started at http://127.6.116.129:34993/ using document root <none> and password file <none>
I20260929 03:29:32.140872 21470 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:32.140938 21470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:32.142234 21470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:32.142904 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.143106 21470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:32.143198 21470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "83d6508750e54fce84b7deac483e3e52"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "69821867aa6692d9044911c418b7e478"
server_key_iv: "8eacb1ce71195fca5f555224a2896a7a"
server_key_version: "encryptionkey@0"
I20260929 03:29:32.143483 21470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:32.167896 21470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:32.170014 21470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:32.170162 21470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:32.170363 21470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:32.170717 21521 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:29:32.172461 21470 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:29:32.172523 21470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:29:32.172569 21470 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:29:32.173142 21470 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:29:32.173198 21470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 03:29:32.173259 21521 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Bootstrap starting.
I20260929 03:29:32.180020 21470 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:39979
I20260929 03:29:32.180115 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:39979 every 8 connection(s)
I20260929 03:29:32.180433 21470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:29:32.183990 21521 log.cc:824] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:32.186426 21521 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:29:32.186712 21521 tablet_bootstrap.cc:492] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Bootstrap complete.
I20260929 03:29:32.187893 21521 ts_tablet_manager.cc:1397] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260929 03:29:32.188431 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21470
I20260929 03:29:32.188671 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:39095
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=45359
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:29:32.189196 21521 raft_consensus.cc:348] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:32.189498 21521 raft_consensus.cc:729] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d6508750e54fce84b7deac483e3e52, State: Initialized, Role: FOLLOWER
I20260929 03:29:32.189623 21521 consensus_queue.cc:260] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:32.189888 21521 ts_tablet_manager.cc:1428] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260929 03:29:32.221423 21631 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:29:32.181016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:42843)
0929 03:29:32.181167 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.181204 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:32.181619 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:32.221278 (+ 39659us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
W20260929 03:29:32.221705 21629 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:41975 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:41975: Client connection negotiation failed: client connection to 127.6.116.190:41975: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:29:32.222045 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.221932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:51087)
0929 03:29:32.221982 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.222028 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:29:32.222357 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.222273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:60303)
0929 03:29:32.222316 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.222338 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:29:32.222462 21629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:29:32.267964 21633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:32.268157 21633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:32.268193 21633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:32.269289 21633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:32.269348 21633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:32.269392 21633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:32.270717 21633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:39095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=45359
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:32.270951 21633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:32.271173 21633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:32.275622 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.275655 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.275657 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:32.275903 21633 server_base.cc:1034] running on GCE node
I20260929 03:29:32.276151 21633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:32.276397 21633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:32.277535 21633 hybrid_clock.cc:648] HybridClock initialized: now 1790652572277529 us; error 27 us; skew 500 ppm
Sep 29 03:29:32 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:32.279873 21633 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:32.280197 21633 webserver.cc:460] Webserver started at http://127.6.116.130:45359/ using document root <none> and password file <none>
I20260929 03:29:32.280366 21633 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:32.280406 21633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:32.281565 21633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:29:32.282202 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.282402 21633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:32.282480 21633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "3b5b88036e094df6b2d3727ae86cdd2d"
format_stamp: "Formatted at 2026-09-29 03:29:30 on dist-test-slave-z2zg"
server_key: "cf85b9b5753a9e0bfae8bc873b8b4c11"
server_key_iv: "8440e03aa8d3c623e515d407835b119e"
server_key_version: "encryptionkey@0"
I20260929 03:29:32.282724 21633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:32.306618 21633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:32.308346 21633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:32.308442 21633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:32.308593 21633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:32.308917 21656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 03:29:32.310091 21656 ts_tablet_manager.cc:1852] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 03:29:32.310305 21656 ts_tablet_manager.cc:1905] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:32.312269 21656 ts_tablet_manager.cc:1918] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:32.312337 21656 log.cc:1196] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:32.312579 21656 ts_tablet_manager.cc:1939] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 3b5b88036e094df6b2d3727ae86cdd2d: Deleting consensus metadata
I20260929 03:29:32.312772 21633 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 03:29:32.312813 21633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 03:29:32.312834 21633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:32.312847 21633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.318207 21633 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:39095
I20260929 03:29:32.318281 21763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:39095 every 8 connection(s)
I20260929 03:29:32.318568 21633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:29:32.319525 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.319168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:52583)
0929 03:29:32.319308 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.319393 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
W20260929 03:29:32.319731 21764 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:41975 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:41975: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
I20260929 03:29:32.319971 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.319905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:44565)
0929 03:29:32.319936 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.319957 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 03:29:32.320197 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.320155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:39807)
0929 03:29:32.320173 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.320188 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260929 03:29:32.320281 21764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:29:32.323583 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21633
I20260929 03:29:32.323793 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:39107
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:41975
--webserver_port=44133
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:32.402797 21767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:32.402961 21767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:32.402983 21767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:32.404074 21767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:32.404111 21767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:32.404136 21767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:32.405542 21767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:39107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=44133
--tserver_master_addrs=127.6.116.190:41975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:32.405762 21767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:32.405978 21767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:32.410470 21772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.410497 21775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:32.410509 21773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:32.410557 21767 server_base.cc:1034] running on GCE node
I20260929 03:29:32.410815 21767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:32.411104 21767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:32.412241 21767 hybrid_clock.cc:648] HybridClock initialized: now 1790652572412234 us; error 35 us; skew 500 ppm
Sep 29 03:29:32 dist-test-slave-z2zg krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:32.414534 21767 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:32.414927 21767 webserver.cc:460] Webserver started at http://127.6.116.131:44133/ using document root <none> and password file <none>
I20260929 03:29:32.415195 21767 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:32.415241 21767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:32.416440 21767 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:29:32.417079 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.417232 21767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:32.417312 21767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "b3b18419996e492ebe4ed291694af6d7"
format_stamp: "Formatted at 2026-09-29 03:29:31 on dist-test-slave-z2zg"
server_key: "6066557ed1b3e2f0e8e752864cb5fa7a"
server_key_iv: "3bfed4ccf136c6fa8529e077ef9ce593"
server_key_version: "encryptionkey@0"
I20260929 03:29:32.417546 21767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:32.427155 21767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:32.428809 21767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:32.428910 21767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:32.429055 21767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:32.429378 21789 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260929 03:29:32.430531 21789 ts_tablet_manager.cc:1852] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 03:29:32.430737 21789 ts_tablet_manager.cc:1905] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:32.432529 21789 ts_tablet_manager.cc:1918] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:32.432611 21789 log.cc:1196] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:32.432912 21789 ts_tablet_manager.cc:1939] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P b3b18419996e492ebe4ed291694af6d7: Deleting consensus metadata
I20260929 03:29:32.433125 21767 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 03:29:32.433195 21767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260929 03:29:32.433236 21767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:32.433265 21767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:32.438937 21767 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:39107
I20260929 03:29:32.439091 21896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:39107 every 8 connection(s)
I20260929 03:29:32.439316 21767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:29:32.440176 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.439860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:39627)
0929 03:29:32.440008 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.440065 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
W20260929 03:29:32.440371 21897 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:41975 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:41975: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
I20260929 03:29:32.440613 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.440531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:48473)
0929 03:29:32.440574 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.440597 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:29:32.440837 21899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:32.440785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:40813)
0929 03:29:32.440810 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:32.440827 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 03:29:32.440912 21897 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:29:32.448215 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21767
I20260929 03:29:33.223016 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.222731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:56611)
0929 03:29:33.222914 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:33.222964 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":51,"threads_started":1}
I20260929 03:29:33.320760 21902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.320583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:45379)
0929 03:29:33.320693 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:33.320725 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":80,"threads_started":1}
I20260929 03:29:33.441365 21903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.441191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:45043)
0929 03:29:33.441299 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:33.441330 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":50,"threads_started":1}
I20260929 03:29:33.736495 21904 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:33.736615 21904 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:33.736999 21904 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
Sep 29 03:29:33 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.131@KRBTEST.COM
Sep 29 03:29:33 dist-test-slave-z2zg krb5kdc[20332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652572, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.130@KRBTEST.COM
I20260929 03:29:33.747561 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.737131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:39095 (local address 127.6.116.129:46577)
0929 03:29:33.737266 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:33.737277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:33.737342 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:33.738207 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:33.738210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:33.738706 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:33.738958 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:33.738964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.740060 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.740060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:33.740842 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:33.740845 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.741433 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.741434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:33.741536 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:33.742148 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:33.742153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:33.744109 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:33.746101 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:33.746103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:33.746349 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:33.746584 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:33.746585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:33.746646 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:33.747011 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:33.747017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:33.747176 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 03:29:33.747286 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:29:33.747471 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":53,"thread_start_us":43,"threads_started":1}
I20260929 03:29:33.747572 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.737260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:50349 (local address 127.6.116.131:39107)
0929 03:29:33.737411 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:33.737422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:33.737450 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:29:33.737961 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:33.737965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:33.738038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:33.738181 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:33.739093 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.739944 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.740940 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.741322 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.741452 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:33.743751 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:33.743754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:33.743755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:33.743764 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:33.745803 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746287 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:33.746289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:33.746290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:33.746416 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746701 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:33.746702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:33.746702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:33.746932 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:33.747028 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:33.747345 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 03:29:33.747461 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":48,"threads_started":1}
I20260929 03:29:33.747608 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.737252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:46577 (local address 127.6.116.130:39095)
0929 03:29:33.737419 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:33.737435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:33.737450 (+ 15us) server_negotiation.cc:407] Connection header received
0929 03:29:33.737993 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:33.737996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:33.738037 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:33.738181 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:33.739093 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.739942 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.740946 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.741337 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.741470 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:33.744214 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:33.744218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:33.744218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:33.744230 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:33.746032 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746407 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:33.746409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:33.746409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:33.746507 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746724 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:33.746725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:33.746726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:33.746934 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:33.747031 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:33.747355 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 03:29:33.747498 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260929 03:29:33.747678 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:33.737131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39107 (local address 127.6.116.129:50349)
0929 03:29:33.737271 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:33.737281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:33.737359 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:33.738207 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:33.738210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:33.738691 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:33.738958 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:33.738964 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.740046 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.740047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:33.740840 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:33.740843 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:33.741425 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:33.741426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:33.741535 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:33.742091 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:33.742098 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:33.743656 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:33.745890 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:33.745895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:33.745897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:33.746190 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:33.746483 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:33.746484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:33.746484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:33.746634 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:33.747011 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:33.747017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:33.747177 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 03:29:33.747286 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:29:33.747548 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":140,"thread_start_us":72,"threads_started":1}
W20260929 03:29:33.748528 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:33.748694 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:33.748740 21516 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:33.748847 21904 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:34.223559 21905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:34.223448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:33611)
0929 03:29:34.223498 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:34.223527 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:29:34.321396 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:34.321212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:36415)
0929 03:29:34.321333 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:34.321361 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260929 03:29:34.441921 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:34.441737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:42035)
0929 03:29:34.441860 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:34.441889 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260929 03:29:35.224153 21911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:35.223956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:37665)
0929 03:29:35.224082 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:35.224111 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260929 03:29:35.321977 21912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:35.321781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:59333)
0929 03:29:35.321912 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:35.321942 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260929 03:29:35.442492 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:35.442311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:59301)
0929 03:29:35.442422 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:35.442457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":73,"threads_started":1}
I20260929 03:29:35.726425 21914 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:35.726514 21914 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:35.726818 21914 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
W20260929 03:29:35.727340 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:35.727433 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:35.727547 21516 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:35.727634 21914 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:36.224733 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:36.224544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:54053)
0929 03:29:36.224663 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:36.224690 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260929 03:29:36.322589 21916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:36.322404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:58155)
0929 03:29:36.322518 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:36.322551 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260929 03:29:36.443135 21917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:36.442844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:43159)
0929 03:29:36.442982 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:36.443020 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260929 03:29:37.225380 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:37.225181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:60051)
0929 03:29:37.225307 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:37.225339 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260929 03:29:37.323308 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:37.323006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:53561)
0929 03:29:37.323240 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:37.323273 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":109,"threads_started":1}
I20260929 03:29:37.443725 21920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:37.443543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:52133)
0929 03:29:37.443662 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:37.443692 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260929 03:29:38.225996 21921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:38.225802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:54813)
0929 03:29:38.225923 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:38.225953 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":56,"threads_started":1}
I20260929 03:29:38.323853 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:38.323685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:60561)
0929 03:29:38.323790 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:38.323819 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260929 03:29:38.444352 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:38.444151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:51755)
0929 03:29:38.444282 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:38.444316 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260929 03:29:38.535233 21924 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:38.535327 21924 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:38.535527 21924 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
W20260929 03:29:38.535923 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:38.535987 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:38.536032 21517 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:38.536149 21924 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:39.226658 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:39.226458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:36897)
0929 03:29:39.226578 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:39.226612 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260929 03:29:39.324394 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:39.324220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:33329)
0929 03:29:39.324327 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:39.324358 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260929 03:29:39.445076 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:39.444882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:35351)
0929 03:29:39.445008 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:39.445041 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":55,"threads_started":1}
I20260929 03:29:40.227344 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:40.227137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:48165)
0929 03:29:40.227269 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:40.227305 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260929 03:29:40.325011 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:40.324832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:45251)
0929 03:29:40.324945 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:40.324975 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260929 03:29:40.445859 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:40.445644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:46615)
0929 03:29:40.445784 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:40.445825 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260929 03:29:41.227911 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:41.227743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:38189)
0929 03:29:41.227846 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:41.227873 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260929 03:29:41.291962 21932 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:41.292030 21932 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:41.292207 21932 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
W20260929 03:29:41.292572 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:41.292644 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:41.292686 21517 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:41.292747 21932 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:41.325623 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:41.325416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:45439)
0929 03:29:41.325539 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:41.325570 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":52,"threads_started":1}
I20260929 03:29:41.446388 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:41.446234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:35169)
0929 03:29:41.446325 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:41.446353 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":51,"threads_started":1}
I20260929 03:29:42.228513 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:42.228319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:35681)
0929 03:29:42.228437 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:42.228469 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260929 03:29:42.326207 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:42.326028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:50369)
0929 03:29:42.326141 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:42.326171 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260929 03:29:42.446899 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:42.446710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:40025)
0929 03:29:42.446836 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:42.446866 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260929 03:29:43.229111 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:43.228904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:59039)
0929 03:29:43.229032 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:43.229065 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260929 03:29:43.326823 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:43.326635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:50925)
0929 03:29:43.326750 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:43.326780 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260929 03:29:43.447626 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:43.447426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:46479)
0929 03:29:43.447543 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:43.447591 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260929 03:29:44.229789 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:44.229570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:48081)
0929 03:29:44.229701 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:44.229733 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260929 03:29:44.327466 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:44.327249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:60349)
0929 03:29:44.327396 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:44.327429 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260929 03:29:44.337469 21943 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:44.337612 21943 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:44.337920 21943 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
W20260929 03:29:44.338181 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:44.338341 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:44.338428 21516 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:44.338511 21943 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:44.448242 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:44.448057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:45273)
0929 03:29:44.448171 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:44.448204 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260929 03:29:45.230422 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:45.230212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:38415)
0929 03:29:45.230349 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:45.230377 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260929 03:29:45.328056 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:45.327876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:56359)
0929 03:29:45.327991 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:45.328020 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260929 03:29:45.448837 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:45.448657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:52443)
0929 03:29:45.448766 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:45.448800 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260929 03:29:46.046818 21948 raft_consensus.cc:482] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:46.046908 21948 raft_consensus.cc:504] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b5b88036e094df6b2d3727ae86cdd2d" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 39095 } } peers { permanent_uuid: "b3b18419996e492ebe4ed291694af6d7" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39107 } } peers { permanent_uuid: "83d6508750e54fce84b7deac483e3e52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39979 } }
I20260929 03:29:46.047132 21948 leader_election.cc:288] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095), b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107)
W20260929 03:29:46.047484 21516 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3b5b88036e094df6b2d3727ae86cdd2d (127.6.116.130:39095): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
W20260929 03:29:46.047555 21517 leader_election.cc:341] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b3b18419996e492ebe4ed291694af6d7 (127.6.116.131:39107): Not found: Tablet not found: 48ccb60bb6e5488e9fa3d2aaa7f3516f
I20260929 03:29:46.047636 21517 leader_election.cc:302] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83d6508750e54fce84b7deac483e3e52; no voters: 3b5b88036e094df6b2d3727ae86cdd2d, b3b18419996e492ebe4ed291694af6d7
I20260929 03:29:46.047711 21948 raft_consensus.cc:2726] T 48ccb60bb6e5488e9fa3d2aaa7f3516f P 83d6508750e54fce84b7deac483e3e52 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:29:46.231114 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:46.230845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:48775)
0929 03:29:46.230989 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:46.231025 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260929 03:29:46.328666 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:46.328451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:50789)
0929 03:29:46.328590 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:46.328622 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260929 03:29:46.449640 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:46.449404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:56531)
0929 03:29:46.449536 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:46.449576 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":101,"threads_started":1}
I20260929 03:29:47.231719 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:47.231533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.129:41061)
0929 03:29:47.231644 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:47.231675 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260929 03:29:47.329288 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:47.329119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.130:34317)
0929 03:29:47.329224 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:47.329253 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260929 03:29:47.450263 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:47.450080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41975 (local address 127.6.116.131:41357)
0929 03:29:47.450199 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:47.450228 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:41975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260929 03:29:47.458676 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21470
I20260929 03:29:47.463388 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21633
I20260929 03:29:47.467633 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21767
2026-09-29T03:29:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18093 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:29:47 dist-test-slave-z2zg krb5kdc[21957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:29:47 dist-test-slave-z2zg krb5kdc[21957](info): set up 2 sockets
Sep 29 03:29:47 dist-test-slave-z2zg krb5kdc[21957](info): commencing operation
krb5kdc: starting...
W20260929 03:29:48.484978 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:29:48 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:29:48.596113 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:48.673964 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:48.674176 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:48.674221 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:48.675401 21970 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:29:48.675464 21970 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:29:48.675485 21970 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:29:48.675504 21970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:29:48.675536 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:48.675562 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:48.675594 21970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:29:48.675626 21970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:29:48.677066 21970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:36383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:48.677340 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:48.677556 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:48.682706 21978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:48.682705 21975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:48.682744 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:48.682721 21970 server_base.cc:1034] running on GCE node
I20260929 03:29:48.683189 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:48.683475 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:48.684639 21970 hybrid_clock.cc:648] HybridClock initialized: now 1790652588684628 us; error 38 us; skew 500 ppm
Sep 29 03:29:48 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:48.687003 21970 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:29:48.687489 21970 webserver.cc:460] Webserver started at http://127.6.116.190:45625/ using document root <none> and password file <none>
I20260929 03:29:48.687714 21970 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:48.687783 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:48.687930 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:48.688905 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "da505c448b8f4377938a74af24a1bd3c"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "c94d5af729dc85ac8fd8f597507a17d3"
server_key_iv: "717a8b7a3ef6d3c6567fff77711af47d"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.689257 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "da505c448b8f4377938a74af24a1bd3c"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "c94d5af729dc85ac8fd8f597507a17d3"
server_key_iv: "717a8b7a3ef6d3c6567fff77711af47d"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.690534 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 03:29:48.691246 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:48.691412 21970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:48.691516 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "da505c448b8f4377938a74af24a1bd3c"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "c94d5af729dc85ac8fd8f597507a17d3"
server_key_iv: "717a8b7a3ef6d3c6567fff77711af47d"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.691587 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:48.705745 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:48.708060 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:48.708221 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:48.712010 21970 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:36383
I20260929 03:29:48.712046 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:36383 every 8 connection(s)
I20260929 03:29:48.712358 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:29:48.713122 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:48.715193 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap starting.
I20260929 03:29:48.715754 22038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:48.716045 22038 log.cc:824] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:48.716681 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: No bootstrap required, opened a new log
I20260929 03:29:48.718147 22038 raft_consensus.cc:348] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:29:48.718269 22038 raft_consensus.cc:374] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:48.718291 22038 raft_consensus.cc:729] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Initialized, Role: FOLLOWER
I20260929 03:29:48.718449 22038 consensus_queue.cc:260] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:29:48.718526 22038 raft_consensus.cc:388] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:29:48.718578 22038 raft_consensus.cc:482] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:29:48.718628 22038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:48.719136 22038 raft_consensus.cc:504] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:29:48.719218 22038 leader_election.cc:302] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da505c448b8f4377938a74af24a1bd3c; no voters:
I20260929 03:29:48.719396 22038 leader_election.cc:288] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:29:48.719444 22041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:48.719662 22038 sys_catalog.cc:564] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:29:48.719897 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 21970
I20260929 03:29:48.720022 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:29:48.720225 6610 external_mini_cluster.cc:1428] Setting key e36770dd03f6af86a5f2dfbd7a503df9
I20260929 03:29:48.719660 22041 raft_consensus.cc:686] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 LEADER]: Becoming Leader. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Running, Role: LEADER
I20260929 03:29:48.720494 22041 consensus_queue.cc:237] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:29:48.721207 22043 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da505c448b8f4377938a74af24a1bd3c. Latest consensus state: current_term: 1 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:29:48.721297 22043 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:48.721748 22042 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:29:48.721836 22042 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:29:48.721982 22052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:29:48.722642 22052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:29:48.724190 22052 catalog_manager.cc:1347] Generated new cluster ID: 5a5ec0f491014c7da5393ddaf1e4d99b
I20260929 03:29:48.724241 22052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:29:48 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:48.731308 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:48.720678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.6.116.190:36383)
0929 03:29:48.721112 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:48.721122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:48.721159 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:29:48.721285 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:48.721293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:48.721352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:48.721512 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:48.722395 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.723432 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.724414 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.725012 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.725129 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:48.727787 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:48.727791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:48.727791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:48.727802 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:48.729674 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:48.730067 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:48.730069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:48.730070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:48.730184 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:48.730426 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:48.730427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:48.730427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:48.730696 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:48.730819 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:48.730966 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 03:29:48.731115 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":62,"threads_started":1}
I20260929 03:29:48.741933 22052 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:29:48.742391 22052 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:29:48.745626 22052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Generated new TSK 0
I20260929 03:29:48.745787 22052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:29:48.775581 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:48.853631 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:48.853848 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:48.853888 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:48.855019 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:48.855099 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:48.855144 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:29:48.856560 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:48.856878 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:48.857115 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:48.861960 22069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:48.861958 22070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:48.861958 22072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:48.862191 22064 server_base.cc:1034] running on GCE node
I20260929 03:29:48.862366 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:48.862574 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:48.863758 22064 hybrid_clock.cc:648] HybridClock initialized: now 1790652588863744 us; error 24 us; skew 500 ppm
Sep 29 03:29:48 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:48.866161 22064 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:29:48.866637 22064 webserver.cc:460] Webserver started at http://127.6.116.129:35831/ using document root <none> and password file <none>
I20260929 03:29:48.866861 22064 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:48.866927 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:48.867121 22064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:48.868237 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "8d744eee3c8746b2959fb242a78de547"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "cdbf43ef63d43697218c25e9c5b6f562"
server_key_iv: "76e5bc3c0aaebd45615f2d3ddfe6ff51"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.868589 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "8d744eee3c8746b2959fb242a78de547"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "cdbf43ef63d43697218c25e9c5b6f562"
server_key_iv: "76e5bc3c0aaebd45615f2d3ddfe6ff51"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.869831 22064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:29:48.870504 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:48.870675 22064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:48.870798 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "8d744eee3c8746b2959fb242a78de547"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "cdbf43ef63d43697218c25e9c5b6f562"
server_key_iv: "76e5bc3c0aaebd45615f2d3ddfe6ff51"
server_key_version: "encryptionkey@0"
I20260929 03:29:48.870869 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:48.881578 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:48.883410 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:48.883558 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:48.883769 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:48.884051 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:48.884104 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:48.884143 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:48.884171 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:48.890045 22064 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:38233
I20260929 03:29:48.890079 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:38233 every 8 connection(s)
I20260929 03:29:48.890372 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
Sep 29 03:29:48 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652588, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:48.898778 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:48.891079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:41833 (local address 127.6.116.190:36383)
0929 03:29:48.891129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:48.891131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:48.891351 (+ 220us) server_negotiation.cc:407] Connection header received
0929 03:29:48.891761 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:48.891761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:48.891770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:48.891820 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:48.892641 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.893150 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.894197 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.894413 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.894475 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:48.896449 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:48.896450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:48.896450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:48.896457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:48.897639 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:48.897988 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:48.897989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:48.897989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:48.898040 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:48.898237 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:48.898238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:48.898238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:48.898340 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:48.898389 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:48.898634 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:29:48.898716 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:29:48.898828 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:48.891038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:41833)
0929 03:29:48.891221 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:48.891258 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:48.891682 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:48.891875 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:48.891878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:48.892391 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:48.892571 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:48.892575 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.893232 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.893233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:48.894089 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:48.894092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:48.894489 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:48.894489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:48.894580 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:48.895089 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:48.895105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:48.896365 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:48.897710 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:48.897715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:48.897717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:48.897923 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:48.898096 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:48.898097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:48.898097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:48.898184 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:48.898393 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:48.898397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:48.898495 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 03:29:48.898573 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:29:48.898677 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260929 03:29:48.899350 22193 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:29:48.899423 22193 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:48.899603 22193 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:29:48.900061 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22064
I20260929 03:29:48.900146 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:29:48.900229 22002 ts_manager.cc:194] Registered new tserver with Master: 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233)
I20260929 03:29:48.900396 6610 external_mini_cluster.cc:1428] Setting key e79569c549fe1cbd0ba60fc3ef9cdf48
I20260929 03:29:48.901196 22002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:41833
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:29:48.925295 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:49.002564 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:49.002782 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:49.002822 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:49.003974 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:49.004041 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:49.004088 22198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:29:49.005492 22198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:49.005731 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:49.005968 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:29:49.010816 22198 server_base.cc:1034] running on GCE node
W20260929 03:29:49.010826 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:49.010798 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:49.010798 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:49.011286 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:49.011464 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:49.012630 22198 hybrid_clock.cc:648] HybridClock initialized: now 1790652589012612 us; error 27 us; skew 500 ppm
Sep 29 03:29:49 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:49.014880 22198 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:29:49.015321 22198 webserver.cc:460] Webserver started at http://127.6.116.130:35985/ using document root <none> and password file <none>
I20260929 03:29:49.015527 22198 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:49.015589 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:49.015729 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:49.016673 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "6e196f8ef52e744f3802d5e65dd688f4"
server_key_iv: "d27d774f593bbcfbf90e4d30a7aad8b5"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.017022 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "6e196f8ef52e744f3802d5e65dd688f4"
server_key_iv: "d27d774f593bbcfbf90e4d30a7aad8b5"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.018240 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:29:49.018940 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.019110 22198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:29:49.019201 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "6e196f8ef52e744f3802d5e65dd688f4"
server_key_iv: "d27d774f593bbcfbf90e4d30a7aad8b5"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.019277 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:49.028180 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:49.029840 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:49.029986 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:49.030189 22198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:49.030457 22198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:49.030508 22198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.030545 22198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:49.030573 22198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.036118 22198 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:36753
I20260929 03:29:49.036183 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:36753 every 8 connection(s)
I20260929 03:29:49.036446 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:29:49.039098 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22198
I20260929 03:29:49.039180 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:29:49.039381 6610 external_mini_cluster.cc:1428] Setting key 443345a4df045e651228ffcc77fca2de
Sep 29 03:29:49 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:49.045732 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.037173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:41437 (local address 127.6.116.190:36383)
0929 03:29:49.037234 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.037236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.037417 (+ 181us) server_negotiation.cc:407] Connection header received
0929 03:29:49.037853 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.037854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.037863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.037916 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:49.038731 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.039357 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.040644 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.040834 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.040880 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.042892 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:49.042894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:49.042894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:49.042902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:49.044383 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.044775 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.044777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.044778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.044841 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.045095 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.045096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.045096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.045212 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:49.045293 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.045549 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.045643 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:29:49.045900 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.037146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:41437)
0929 03:29:49.037305 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.037341 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.037779 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.037961 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.037964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.038469 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:49.038662 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.038666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.039441 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.039442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.040556 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.040559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.040905 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.040906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.041038 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.041475 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:49.041486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:49.042799 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:49.044468 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:49.044474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:49.044476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:49.044690 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:49.044917 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:49.044917 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:49.044918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:49.045022 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:49.045284 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:49.045290 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:49.045421 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.045525 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.045666 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260929 03:29:49.046337 22327 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:29:49.046408 22327 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:49.046589 22327 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:29:49.046959 22002 ts_manager.cc:194] Registered new tserver with Master: 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:29:49.047369 22002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:41437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:29:49.063570 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:29:49.140867 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:29:49.141057 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:29:49.141079 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:29:49.142117 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:29:49.142148 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:29:49.142172 22332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:29:49.143450 22332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:29:49.143612 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:29:49.143759 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:29:49.148329 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:49.148352 22332 server_base.cc:1034] running on GCE node
W20260929 03:29:49.148469 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:29:49.148640 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:29:49.149665 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:29:49.149945 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:29:49.151116 22332 hybrid_clock.cc:648] HybridClock initialized: now 1790652589151092 us; error 53 us; skew 500 ppm
Sep 29 03:29:49 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:29:49.153501 22332 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:29:49.153880 22332 webserver.cc:460] Webserver started at http://127.6.116.131:38941/ using document root <none> and password file <none>
I20260929 03:29:49.154099 22332 fs_manager.cc:359] Metadata directory not provided
I20260929 03:29:49.154162 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:29:49.154301 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:29:49.155290 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "1a3bd670ea55f5285500ab3811e8e1f3"
server_key_iv: "d1c24cdbe0f241f63877c8b78a39085c"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.155622 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "1a3bd670ea55f5285500ab3811e8e1f3"
server_key_iv: "d1c24cdbe0f241f63877c8b78a39085c"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.156845 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:29:49.157505 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.157665 22332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:29:49.157754 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "1a3bd670ea55f5285500ab3811e8e1f3"
server_key_iv: "d1c24cdbe0f241f63877c8b78a39085c"
server_key_version: "encryptionkey@0"
I20260929 03:29:49.157830 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:29:49.174024 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:29:49.176105 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:29:49.176259 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:29:49.176465 22332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:29:49.176731 22332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:29:49.176781 22332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.176813 22332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:29:49.176846 22332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:29:49.182291 22332 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:38945
I20260929 03:29:49.182360 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:38945 every 8 connection(s)
I20260929 03:29:49.182631 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:29:49.187531 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22332
I20260929 03:29:49.187636 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:29:49.187812 6610 external_mini_cluster.cc:1428] Setting key 3011fc5ac07fdf027f2a81123bc2cbd9
Sep 29 03:29:49 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652589, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:29:49.191111 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.183296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:53355 (local address 127.6.116.190:36383)
0929 03:29:49.183356 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.183358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.183541 (+ 183us) server_negotiation.cc:407] Connection header received
0929 03:29:49.183983 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.183984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.183996 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.184057 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:49.184903 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.185378 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.186341 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.186550 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.186633 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.188769 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:49.188770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:49.188771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:49.188778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:49.189808 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.190184 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.190185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.190185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.190239 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.190492 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.190493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.190494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.190603 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:49.190675 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.190920 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.190993 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 03:29:49.191349 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.183272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:53355)
0929 03:29:49.183422 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.183460 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.183883 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.184117 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.184120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.184623 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:29:49.184780 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.184784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.185467 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.185468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.186245 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.186248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.186643 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.186644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.186765 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.187193 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:29:49.187201 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:29:49.188681 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:29:49.189903 (+ 1222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:49.189909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:49.189910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:49.190111 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:49.190312 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:29:49.190312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:29:49.190313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:29:49.190420 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:29:49.190667 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:29:49.190672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:29:49.190797 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.190899 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.191034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260929 03:29:49.191742 22461 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:29:49.191808 22461 heartbeater.cc:464] Registering TS with master...
I20260929 03:29:49.191924 22461 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:29:49.192215 22002 ts_manager.cc:194] Registered new tserver with Master: 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945)
I20260929 03:29:49.192554 22002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:53355
I20260929 03:29:49.199249 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:29:49.205442 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.200464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.6.116.190:36383)
0929 03:29:49.200513 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.200515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.200598 (+ 83us) server_negotiation.cc:407] Connection header received
0929 03:29:49.200639 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.200640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.200646 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.200705 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:29:49.201182 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.201590 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.202240 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.202422 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.202467 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.203505 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:29:49.203506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:29:49.203506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:29:49.203511 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:29:49.204560 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.204869 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.204870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.204870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.204913 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:29:49.205042 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:29:49.205043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:29:49.205043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:29:49.205155 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:29:49.205223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.205333 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.205391 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:29:49.206756 22002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:29:49.208426 22002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:29:49.214512 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38945 (local address 127.0.0.1:53818)
0929 03:29:49.210780 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.210792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.210848 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.211145 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.211146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.211148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:49.211200 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.211202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.212374 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.212374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.213418 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.213419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.214248 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.214249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.214326 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.214339 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.214384 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.214437 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260929 03:29:49.214573 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.6.116.130:36753)
0929 03:29:49.210608 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.210612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.210885 (+ 273us) server_negotiation.cc:407] Connection header received
0929 03:29:49.210926 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.210928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.210943 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.211020 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:49.211338 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.212051 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.213486 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.214148 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.214219 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.214272 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.214450 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.214496 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260929 03:29:49.214761 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.6.116.131:38945)
0929 03:29:49.210939 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.210943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.210950 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:29:49.210984 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.210985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.211017 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.211145 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:49.211286 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.212282 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.213486 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.214148 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.214551 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.214610 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.214637 (+ 27us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.214682 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260929 03:29:49.214514 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.0.0.1:42182)
0929 03:29:49.210685 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.210720 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.210789 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.211223 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.211224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.211226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:49.211260 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.211261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.212159 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.212160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.213415 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.213418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.214300 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.214300 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.214364 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.214368 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.214419 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.214449 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":63,"threads_started":1}
I20260929 03:29:49.215320 22395 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5328952fdf4655938538b13f0c9d0b (DEFAULT_TABLE table=test-table [id=84a2285070c342e096ac8daf173f12c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:49.215342 22261 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5328952fdf4655938538b13f0c9d0b (DEFAULT_TABLE table=test-table [id=84a2285070c342e096ac8daf173f12c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:49.215656 22395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5328952fdf4655938538b13f0c9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:49.215658 22261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5328952fdf4655938538b13f0c9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:49.217890 22475 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Bootstrap starting.
I20260929 03:29:49.218154 22476 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Bootstrap starting.
I20260929 03:29:49.218549 22475 tablet_bootstrap.cc:654] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:49.218714 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:38233 (local address 127.0.0.1:44024)
0929 03:29:49.210759 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.210768 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.210813 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.211062 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.211065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.211067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:49.211127 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.211129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.213512 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.213513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.214141 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.214142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.218461 (+ 4319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.218462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.218530 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.218534 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.218587 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.218634 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":61,"threads_started":1}
I20260929 03:29:49.218976 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.210553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.6.116.129:38233)
0929 03:29:49.210708 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.210711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.210882 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:29:49.210926 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.210928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.210943 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.211018 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:49.211200 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.212273 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.214226 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.218352 (+ 4126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.218461 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.218520 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.218816 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.218863 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260929 03:29:49.218976 22476 tablet_bootstrap.cc:654] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:49.219318 22475 log.cc:824] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:49.219606 22476 log.cc:824] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:49.219651 22127 tablet_service.cc:1467] Processing CreateTablet for tablet 2f5328952fdf4655938538b13f0c9d0b (DEFAULT_TABLE table=test-table [id=84a2285070c342e096ac8daf173f12c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:29:49.219931 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f5328952fdf4655938538b13f0c9d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:29:49.220016 22475 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: No bootstrap required, opened a new log
I20260929 03:29:49.220077 22475 ts_tablet_manager.cc:1397] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:29:49.221650 22475 raft_consensus.cc:348] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.221859 22475 raft_consensus.cc:374] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:49.221941 22475 raft_consensus.cc:729] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a2e3edae5c4d839c33b9ccf07d31d5, State: Initialized, Role: FOLLOWER
I20260929 03:29:49.222057 22475 consensus_queue.cc:260] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.222324 22475 ts_tablet_manager.cc:1428] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:29:49.222321 22461 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
I20260929 03:29:49.222750 22480 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Bootstrap starting.
I20260929 03:29:49.223387 22480 tablet_bootstrap.cc:654] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Neither blocks nor log segments found. Creating new log.
I20260929 03:29:49.223419 22476 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: No bootstrap required, opened a new log
I20260929 03:29:49.223479 22476 ts_tablet_manager.cc:1397] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 03:29:49.223862 22480 log.cc:824] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Log is configured to *not* fsync() on all Append() calls
I20260929 03:29:49.224519 22480 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: No bootstrap required, opened a new log
I20260929 03:29:49.224594 22480 ts_tablet_manager.cc:1397] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:29:49.224697 22476 raft_consensus.cc:348] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.224813 22476 raft_consensus.cc:374] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:49.224833 22476 raft_consensus.cc:729] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9ea0dcb533458c9b22b515b4f065a6, State: Initialized, Role: FOLLOWER
I20260929 03:29:49.224910 22476 consensus_queue.cc:260] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.225081 22476 ts_tablet_manager.cc:1428] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:29:49.225167 22327 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
I20260929 03:29:49.226260 22480 raft_consensus.cc:348] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.226404 22480 raft_consensus.cc:374] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:29:49.226425 22480 raft_consensus.cc:729] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d744eee3c8746b2959fb242a78de547, State: Initialized, Role: FOLLOWER
I20260929 03:29:49.226524 22480 consensus_queue.cc:260] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.226729 22480 ts_tablet_manager.cc:1428] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:29:49.226814 22193 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
W20260929 03:29:49.286978 22328 tablet.cc:2369] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 03:29:49.391058 22194 tablet.cc:2369] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:29:49.430451 22483 raft_consensus.cc:482] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:29:49.430555 22483 raft_consensus.cc:504] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.430951 22483 leader_election.cc:288] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753), 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945)
W20260929 03:29:49.433192 22462 tablet.cc:2369] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:29:49.433995 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.431229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:43855 (local address 127.6.116.130:36753)
0929 03:29:49.431294 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.431296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.431365 (+ 69us) server_negotiation.cc:407] Connection header received
0929 03:29:49.431482 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.431484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.431496 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.431541 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:49.431709 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.432193 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433174 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.433646 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433729 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.433739 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.433890 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.433928 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:29:49.434046 22474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.431461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:51565 (local address 127.6.116.131:38945)
0929 03:29:49.431501 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.431503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.431511 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:29:49.431558 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.431559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.431568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.431642 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:29:49.431802 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.432268 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433214 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.433690 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433778 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.433786 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.433944 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.433983 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:29:49.434068 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.431338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38945 (local address 127.6.116.129:51565)
0929 03:29:49.431420 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.431431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.431496 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.431649 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.431650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.431651 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:49.431732 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.431734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.432356 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.432357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.433122 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.433123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.433798 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.433870 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.433873 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.433938 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.433975 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":40,"threads_started":1}
I20260929 03:29:49.434219 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.431166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.6.116.129:43855)
0929 03:29:49.431291 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.431302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:29:49.431370 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:29:49.431554 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:29:49.431555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:29:49.431557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:29:49.431636 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.431638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.432284 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.432285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.433075 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:29:49.433077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.433741 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.433742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:29:49.433818 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.433821 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:29:49.433885 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:29:49.433925 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260929 03:29:49.434373 22415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5328952fdf4655938538b13f0c9d0b" candidate_uuid: "8d744eee3c8746b2959fb242a78de547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" is_pre_election: true
I20260929 03:29:49.434520 22415 raft_consensus.cc:2454] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d744eee3c8746b2959fb242a78de547 in term 0.
I20260929 03:29:49.434760 22082 leader_election.cc:302] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a2e3edae5c4d839c33b9ccf07d31d5, 8d744eee3c8746b2959fb242a78de547; no voters:
I20260929 03:29:49.434862 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5328952fdf4655938538b13f0c9d0b" candidate_uuid: "8d744eee3c8746b2959fb242a78de547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" is_pre_election: true
I20260929 03:29:49.435014 22281 raft_consensus.cc:2454] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d744eee3c8746b2959fb242a78de547 in term 0.
I20260929 03:29:49.435091 22483 raft_consensus.cc:2781] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:29:49.435153 22483 raft_consensus.cc:482] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:29:49.435184 22483 raft_consensus.cc:3037] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:49.435894 22483 raft_consensus.cc:504] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.435981 22483 leader_election.cc:288] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753), 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945)
I20260929 03:29:49.436175 22415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5328952fdf4655938538b13f0c9d0b" candidate_uuid: "8d744eee3c8746b2959fb242a78de547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
I20260929 03:29:49.436244 22415 raft_consensus.cc:3037] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:49.436936 22415 raft_consensus.cc:2454] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d744eee3c8746b2959fb242a78de547 in term 1.
I20260929 03:29:49.437058 22281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f5328952fdf4655938538b13f0c9d0b" candidate_uuid: "8d744eee3c8746b2959fb242a78de547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
I20260929 03:29:49.437104 22082 leader_election.cc:302] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a2e3edae5c4d839c33b9ccf07d31d5, 8d744eee3c8746b2959fb242a78de547; no voters:
I20260929 03:29:49.437136 22281 raft_consensus.cc:3037] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:29:49.437196 22483 raft_consensus.cc:2781] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:29:49.437376 22483 raft_consensus.cc:686] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 LEADER]: Becoming Leader. State: Replica: 8d744eee3c8746b2959fb242a78de547, State: Running, Role: LEADER
I20260929 03:29:49.437469 22483 consensus_queue.cc:237] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:29:49.437984 22281 raft_consensus.cc:2454] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d744eee3c8746b2959fb242a78de547 in term 1.
I20260929 03:29:49.438203 22002 catalog_manager.cc:5495] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d744eee3c8746b2959fb242a78de547 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "8d744eee3c8746b2959fb242a78de547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } health_report { overall_health: HEALTHY } } }
I20260929 03:29:49.494910 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.492712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.6.116.129:38233)
0929 03:29:49.492748 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:29:49.492749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:29:49.492755 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:29:49.492829 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:29:49.492830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:29:49.492845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:29:49.492931 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:29:49.493022 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.493487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.494176 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:29:49.494370 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:29:49.494418 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:29:49.494608 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:29:49.494695 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:29:49.494755 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:29:49.494804 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 03:29:49.494819 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:29:49.496464 22415 raft_consensus.cc:1261] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Refusing update from remote peer 8d744eee3c8746b2959fb242a78de547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:49.496515 22281 raft_consensus.cc:1261] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Refusing update from remote peer 8d744eee3c8746b2959fb242a78de547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:29:49.496771 22483 consensus_queue.cc:1035] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:49.496902 22486 consensus_queue.cc:1035] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:29:49.498941 22489 mvcc.cc:205] Tried to move back new op lower bound from 7334513006575190016 to 7334513006336434176. Current Snapshot: MvccSnapshot[applied={T|T < 7334513006575190016}]
I20260929 03:29:49.501720 22001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:49.501812 22001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:29:49.502490 22001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Sending DeleteTablet for 3 replicas of tablet 2f5328952fdf4655938538b13f0c9d0b
I20260929 03:29:49.502763 22127 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5328952fdf4655938538b13f0c9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:49 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44024
I20260929 03:29:49.502789 22261 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5328952fdf4655938538b13f0c9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:49 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:42182
I20260929 03:29:49.502799 22395 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5328952fdf4655938538b13f0c9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:49 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53818
I20260929 03:29:49.502982 22499 tablet_replica.cc:330] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: stopping tablet replica
I20260929 03:29:49.503026 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22064
I20260929 03:29:49.503091 22499 raft_consensus.cc:2229] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:29:49.504035 22501 tablet_replica.cc:330] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: stopping tablet replica
I20260929 03:29:49.504143 22501 raft_consensus.cc:2229] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:49.504240 22501 raft_consensus.cc:2258] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:49.504606 22500 tablet_replica.cc:330] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: stopping tablet replica
I20260929 03:29:49.504611 22501 ts_tablet_manager.cc:1905] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:29:49.504699 22500 raft_consensus.cc:2229] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:29:49.504788 22500 raft_consensus.cc:2258] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:29:49.506007 22501 ts_tablet_manager.cc:1918] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:49.506088 22501 log.cc:1196] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/2f5328952fdf4655938538b13f0c9d0b
I20260929 03:29:49.506354 22501 ts_tablet_manager.cc:1939] T 2f5328952fdf4655938538b13f0c9d0b P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting consensus metadata
I20260929 03:29:49.506729 21988 catalog_manager.cc:4841] TS 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): tablet 2f5328952fdf4655938538b13f0c9d0b (table test-table [id=84a2285070c342e096ac8daf173f12c2]) successfully deleted
I20260929 03:29:49.507349 22500 ts_tablet_manager.cc:1905] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:29:49.508469 21986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:38233 (error 108)
I20260929 03:29:49.508621 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22198
I20260929 03:29:49.508699 22500 ts_tablet_manager.cc:1918] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:29:49.508769 22500 log.cc:1196] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/2f5328952fdf4655938538b13f0c9d0b
I20260929 03:29:49.508948 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.508826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:38233 (local address 127.0.0.1:44036)
0929 03:29:49.508884 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.508912 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:29:49.508996 22500 ts_tablet_manager.cc:1939] T 2f5328952fdf4655938538b13f0c9d0b P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting consensus metadata
W20260929 03:29:49.509253 21986 catalog_manager.cc:4584] TS 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f5328952fdf4655938538b13f0c9d0b: Network error: Client connection negotiation failed: client connection to 127.6.116.129:38233: connect: Connection refused (error 111)
I20260929 03:29:49.512981 22470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:29:49.512862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.0.0.1:42198)
0929 03:29:49.512926 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 03:29:49.512954 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:36753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:29:49.512976 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22332
I20260929 03:29:49.517498 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 21970
I20260929 03:30:09.522980 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--webserver_interface=127.6.116.190
--webserver_port=45625
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:09.598589 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:09.598769 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:09.598790 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:09.599970 22503 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:30:09.600010 22503 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:30:09.600024 22503 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:30:09.600035 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:09.600047 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:09.600060 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:09.600126 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:09.600147 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:09.601550 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:36383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:09.601732 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:09.601869 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:09.609861 22510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:09.609908 22509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.609994 22503 server_base.cc:1034] running on GCE node
W20260929 03:30:09.609926 22512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.610374 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:09.610705 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:09.611866 22503 hybrid_clock.cc:648] HybridClock initialized: now 1790652609611852 us; error 25 us; skew 500 ppm
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:09.614287 22503 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:30:09.614724 22503 webserver.cc:460] Webserver started at http://127.6.116.190:45625/ using document root <none> and password file <none>
I20260929 03:30:09.614967 22503 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:09.615046 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:09.616349 22503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:30:09.616994 22519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:09.617174 22503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:09.617273 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "da505c448b8f4377938a74af24a1bd3c"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "c94d5af729dc85ac8fd8f597507a17d3"
server_key_iv: "717a8b7a3ef6d3c6567fff77711af47d"
server_key_version: "encryptionkey@0"
I20260929 03:30:09.617630 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:09.628808 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:09.630944 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:09.631111 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:09.634670 22503 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:36383
I20260929 03:30:09.634698 22571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:36383 every 8 connection(s)
I20260929 03:30:09.635075 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:09.636356 22572 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:30:09.636957 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap starting.
I20260929 03:30:09.637199 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22503
I20260929 03:30:09.637476 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:38233
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=35831
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:30:09.645036 22572 log.cc:824] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:09.646697 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:30:09.646970 22572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap complete.
I20260929 03:30:09.649088 22572 raft_consensus.cc:348] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:09.649322 22572 raft_consensus.cc:729] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Initialized, Role: FOLLOWER
I20260929 03:30:09.649488 22572 consensus_queue.cc:260] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:09.649559 22572 raft_consensus.cc:388] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:09.649619 22572 raft_consensus.cc:482] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:09.649685 22572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 1 FOLLOWER]: Advancing to term 2
I20260929 03:30:09.650632 22572 raft_consensus.cc:504] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:09.650753 22572 leader_election.cc:302] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da505c448b8f4377938a74af24a1bd3c; no voters:
I20260929 03:30:09.650941 22572 leader_election.cc:288] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 03:30:09.650970 22576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Leader election won for term 2
I20260929 03:30:09.651099 22576 raft_consensus.cc:686] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 LEADER]: Becoming Leader. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Running, Role: LEADER
I20260929 03:30:09.651191 22576 consensus_queue.cc:237] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:09.651216 22572 sys_catalog.cc:564] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:09.651436 22578 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da505c448b8f4377938a74af24a1bd3c. Latest consensus state: current_term: 2 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:30:09.651530 22578 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:09.651695 22577 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:30:09.651746 22580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:09.651752 22577 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:09.652640 22580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:09.652796 22580 catalog_manager.cc:1259] Loaded cluster ID: 5a5ec0f491014c7da5393ddaf1e4d99b
I20260929 03:30:09.652958 22580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:09.653784 22580 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:09.653972 22580 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Loaded TSK: 0
I20260929 03:30:09.661280 22580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Generated new TSK 1
I20260929 03:30:09.661367 22580 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:30:09.741669 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:09.741914 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:09.741945 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:09.743562 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:09.743618 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:09.743655 22574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:09.745612 22574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:38233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=35831
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:09.745841 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:09.746078 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:09.751008 22599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:09.751014 22600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.751226 22574 server_base.cc:1034] running on GCE node
W20260929 03:30:09.751304 22602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.751560 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:09.751893 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:09.753046 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790652609753032 us; error 29 us; skew 500 ppm
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:09.755563 22574 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:30:09.756070 22574 webserver.cc:460] Webserver started at http://127.6.116.129:35831/ using document root <none> and password file <none>
I20260929 03:30:09.756319 22574 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:09.756400 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:09.758077 22574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:09.758855 22609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:09.759117 22574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:09.759213 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "8d744eee3c8746b2959fb242a78de547"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "cdbf43ef63d43697218c25e9c5b6f562"
server_key_iv: "76e5bc3c0aaebd45615f2d3ddfe6ff51"
server_key_version: "encryptionkey@0"
I20260929 03:30:09.759595 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:09.773736 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:09.776114 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:09.776274 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:09.776482 22574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:09.776932 22616 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:30:09.778265 22574 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:30:09.778319 22574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:30:09.778350 22574 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:30:09.779142 22574 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:30:09.779184 22574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:09.779256 22616 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Bootstrap starting.
I20260929 03:30:09.786230 22574 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:38233
I20260929 03:30:09.786350 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:38233 every 8 connection(s)
I20260929 03:30:09.786660 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:09.789949 22616 log.cc:824] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:09.792393 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22574
I20260929 03:30:09.792403 22616 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:30:09.792692 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:36753
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=35985
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:30:09.792723 22616 tablet_bootstrap.cc:492] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Bootstrap complete.
I20260929 03:30:09.794104 22616 ts_tablet_manager.cc:1397] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260929 03:30:09.795372 22616 raft_consensus.cc:348] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:09.795625 22616 raft_consensus.cc:729] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d744eee3c8746b2959fb242a78de547, State: Initialized, Role: FOLLOWER
I20260929 03:30:09.795780 22616 consensus_queue.cc:260] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:09.796048 22616 ts_tablet_manager.cc:1428] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:09.801100 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.787426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:51555)
0929 03:30:09.787612 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:09.787661 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:09.788349 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:09.790668 (+ 2319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:09.790673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:09.791006 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:09.791278 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.791285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.792905 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.792906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.794314 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.794317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.795158 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.795159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.795279 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.795831 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:09.795840 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:09.797303 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:09.799367 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.799373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.799375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.799660 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.799922 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.799923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.799924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.800031 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.800438 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:09.800444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:09.800589 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 03:30:09.800698 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 03:30:09.800869 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260929 03:30:09.801113 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.787527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:51555 (local address 127.6.116.190:36383)
0929 03:30:09.790245 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:09.790254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:09.790293 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:30:09.790390 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:09.790395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:09.790462 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:09.790604 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:09.791406 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.792751 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.794417 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.795002 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.795127 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.797431 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:09.797434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:09.797434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:09.797443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:09.799273 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.799764 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.799767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.799767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.799852 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.800096 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.800097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.800097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.800366 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:09.800458 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:09.800780 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 03:30:09.800912 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":77,"threads_started":1}
I20260929 03:30:09.802122 22724 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:30:09.802214 22724 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:09.802376 22724 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:30:09.803206 22536 ts_manager.cc:194] Registered new tserver with Master: 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233)
I20260929 03:30:09.804378 22536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:51555
I20260929 03:30:09.805328 22724 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.129@KRBTEST.COM
I20260929 03:30:09.814311 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.803965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.6.116.129:38233)
0929 03:30:09.804153 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:09.804157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:09.804166 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:09.804194 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:09.804196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:09.804209 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:09.805732 (+ 1523us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:09.805778 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.806851 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.808197 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.808679 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.808757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.811236 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:09.811243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:09.811243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:09.811250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:09.813024 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.813447 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.813449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.813450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.813508 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.813739 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.813740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.813740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.813876 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:09.813953 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:09.814140 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:30:09.814219 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260929 03:30:09.814311 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.803789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:38233 (local address 127.0.0.1:53678)
0929 03:30:09.803936 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:09.803972 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:09.804034 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:09.804297 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:09.804301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:09.805542 (+ 1241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:09.805645 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.805647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.806944 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.806946 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.808097 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.808101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.808792 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.808793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.808896 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.809404 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:09.809410 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:09.811154 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:09.813110 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.813114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.813116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.813363 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.813581 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.813581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.813582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.813667 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.813953 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:09.813958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:09.814010 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:30:09.814095 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 03:30:09.814201 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260929 03:30:09.815294 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f5328952fdf4655938538b13f0c9d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:29:49 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53678
I20260929 03:30:09.815840 22733 tablet_replica.cc:330] stopping tablet replica
I20260929 03:30:09.815943 22733 raft_consensus.cc:2229] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:09.816004 22733 raft_consensus.cc:2258] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:09.816277 22733 ts_tablet_manager.cc:1905] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:09.818696 22733 ts_tablet_manager.cc:1918] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:09.818765 22733 log.cc:1196] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal/wals/2f5328952fdf4655938538b13f0c9d0b
I20260929 03:30:09.818993 22733 ts_tablet_manager.cc:1939] T 2f5328952fdf4655938538b13f0c9d0b P 8d744eee3c8746b2959fb242a78de547: Deleting consensus metadata
I20260929 03:30:09.819500 22520 catalog_manager.cc:4841] TS 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233): tablet 2f5328952fdf4655938538b13f0c9d0b (table test-table [id=84a2285070c342e096ac8daf173f12c2]) successfully deleted
W20260929 03:30:09.881516 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:09.881759 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:09.881800 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:09.882893 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:09.882959 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:09.883001 22729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:09.884420 22729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=35985
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:09.884667 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:09.884886 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:09.889694 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:09.889681 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.889839 22729 server_base.cc:1034] running on GCE node
W20260929 03:30:09.889678 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:09.890101 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:09.890331 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:09.891484 22729 hybrid_clock.cc:648] HybridClock initialized: now 1790652609891478 us; error 38 us; skew 500 ppm
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:09.893720 22729 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:30:09.894145 22729 webserver.cc:460] Webserver started at http://127.6.116.130:35985/ using document root <none> and password file <none>
I20260929 03:30:09.894358 22729 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:09.894438 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:09.895752 22729 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260929 03:30:09.896284 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:09.896435 22729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:09.896526 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "6e196f8ef52e744f3802d5e65dd688f4"
server_key_iv: "d27d774f593bbcfbf90e4d30a7aad8b5"
server_key_version: "encryptionkey@0"
I20260929 03:30:09.896816 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:09.909902 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:09.911547 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:09.911712 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:09.911967 22729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:09.912266 22729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:09.912324 22729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:09.912364 22729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:09.912407 22729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:09.917949 22729 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:36753
I20260929 03:30:09.917994 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:36753 every 8 connection(s)
I20260929 03:30:09.918325 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:30:09 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652609, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:09.923900 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22729
I20260929 03:30:09.924126 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:38945
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=38941
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:30:09.927451 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.918906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:36037 (local address 127.6.116.190:36383)
0929 03:30:09.918967 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:09.918970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:09.919206 (+ 236us) server_negotiation.cc:407] Connection header received
0929 03:30:09.919654 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:09.919654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:09.919666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:09.919727 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:09.920553 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.921138 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.922063 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.922260 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.922310 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.924434 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:09.924436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:09.924436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:09.924444 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:09.926100 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.926470 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.926471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.926471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.926532 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:09.926790 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:09.926791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:09.926791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:09.926899 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:09.926973 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:09.927281 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 03:30:09.927365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:30:09.927707 22864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:09.918900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:36037)
0929 03:30:09.919075 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:09.919113 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:09.919572 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:09.919792 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:09.919795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:09.920318 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:09.920481 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.920485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.921254 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.921254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.921997 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:09.921999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:09.922330 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:09.922331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:09.922423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:09.922821 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:09.922828 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:09.924335 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:09.926179 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.926185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.926187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.926392 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.926604 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:09.926605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:09.926605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:09.926708 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:09.926965 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:09.926971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:09.927152 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 03:30:09.927260 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:30:09.927397 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260929 03:30:09.928164 22862 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:30:09.928246 22862 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:09.928404 22862 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:30:09.928778 22536 ts_manager.cc:194] Registered new tserver with Master: 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
I20260929 03:30:09.929324 22536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:36037
W20260929 03:30:10.002604 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:10.002780 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:10.002803 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:10.003937 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:10.003995 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:10.004020 22865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:10.005330 22865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:38945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=38941
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:10.005518 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:10.005686 22865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:10.010411 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:10.010423 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:10.010423 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:10.010566 22865 server_base.cc:1034] running on GCE node
I20260929 03:30:10.010747 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:10.010974 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:10.012128 22865 hybrid_clock.cc:648] HybridClock initialized: now 1790652610012109 us; error 30 us; skew 500 ppm
Sep 29 03:30:10 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652610, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:10.014547 22865 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:30:10.015007 22865 webserver.cc:460] Webserver started at http://127.6.116.131:38941/ using document root <none> and password file <none>
I20260929 03:30:10.015304 22865 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:10.015381 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:10.016605 22865 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:10.017099 22880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:10.017277 22865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:10.017343 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "1a3bd670ea55f5285500ab3811e8e1f3"
server_key_iv: "d1c24cdbe0f241f63877c8b78a39085c"
server_key_version: "encryptionkey@0"
I20260929 03:30:10.017588 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:10.034337 22865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:10.036490 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:10.036594 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:10.036746 22865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:10.037035 22865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:10.037068 22865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:10.037088 22865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:10.037135 22865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:10.042764 22865 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:38945
I20260929 03:30:10.042812 22993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:38945 every 8 connection(s)
I20260929 03:30:10.043112 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:30:10 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652610, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:10.049244 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 22865
I20260929 03:30:10.052561 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.043746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:33191 (local address 127.6.116.190:36383)
0929 03:30:10.043806 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.043811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.044003 (+ 192us) server_negotiation.cc:407] Connection header received
0929 03:30:10.044489 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.044491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.044502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.044553 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:10.045402 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.045883 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.046868 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.047126 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.047231 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.049198 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:10.049199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:10.049200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:10.049207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:10.050964 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.051487 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.051488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.051488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.051538 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.051801 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.051801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.051801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.051891 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:10.051939 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.052395 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.052475 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:30:10.052608 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.043736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:33191)
0929 03:30:10.043891 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.043929 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:10.044389 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:10.044614 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:10.044618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:10.045141 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:10.045306 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.045310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.045974 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.045975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.046763 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.046765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.047230 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.047231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.047368 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.047782 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:10.047792 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:10.049107 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:10.051091 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:10.051097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:10.051099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:10.051360 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:10.051600 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:10.051601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:10.051601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:10.051738 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:10.051954 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:10.051959 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:10.052111 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 03:30:10.052216 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 03:30:10.052401 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260929 03:30:10.052990 22994 heartbeater.cc:347] Connected to a master server at 127.6.116.190:36383
I20260929 03:30:10.053114 22994 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:10.053297 22994 heartbeater.cc:511] Master 127.6.116.190:36383 requested a full tablet report, sending...
I20260929 03:30:10.053687 22536 ts_manager.cc:194] Registered new tserver with Master: 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945)
I20260929 03:30:10.054113 22536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:33191
I20260929 03:30:10.056418 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.049646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.6.116.190:36383)
0929 03:30:10.049766 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.049768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.049865 (+ 97us) server_negotiation.cc:407] Connection header received
0929 03:30:10.049907 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.049908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.049920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.049990 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:10.050554 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.051267 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.052331 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.052595 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.052731 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.054020 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:10.054021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:10.054022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:10.054028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:10.055551 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.055865 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.055865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.055866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.055908 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.056051 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.056052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.056052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.056139 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:10.056242 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.056297 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.056354 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260929 03:30:10.057088 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:10.059453 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.057439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.6.116.190:36383)
0929 03:30:10.057472 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.057474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.057545 (+ 71us) server_negotiation.cc:407] Connection header received
0929 03:30:10.057609 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.057610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.057634 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.057691 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:10.057753 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.058182 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.058862 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.059038 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.059134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.059271 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:10.059304 (+ 33us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 03:30:10.059415 (+ 111us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49820: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:30:10.059487 22998 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49820: token expired
W20260929 03:30:10.059602 22466 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.116.190:36383)
I20260929 03:30:10.064862 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.059795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.6.116.190:36383)
0929 03:30:10.059829 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.059831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.059839 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:10.059883 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.059883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.059891 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.059948 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:10.060367 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.060778 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.061401 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.061555 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.061598 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.062636 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:10.062636 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:10.062636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:10.062640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:10.063861 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.064254 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.064255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.064255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.064300 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:10.064468 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:10.064469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:10.064469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:10.064568 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:10.064639 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.064756 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.064811 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:30:10.065853 22536 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:10.067509 22536 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:10.069761 22652 tablet_service.cc:1467] Processing CreateTablet for tablet 8251887d147043bfb974c1be07cf9af4 (DEFAULT_TABLE table=test-table [id=4d32c94b8a324c2c9457cef60c0e15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:10.069964 22652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251887d147043bfb974c1be07cf9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:10.071471 22616 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Bootstrap starting.
I20260929 03:30:10.071806 22616 tablet_bootstrap.cc:654] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:10.072624 22616 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: No bootstrap required, opened a new log
I20260929 03:30:10.072676 22616 ts_tablet_manager.cc:1397] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:10.072983 22616 raft_consensus.cc:348] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.073051 22616 raft_consensus.cc:374] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:10.073096 22616 raft_consensus.cc:729] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d744eee3c8746b2959fb242a78de547, State: Initialized, Role: FOLLOWER
I20260929 03:30:10.073148 22616 consensus_queue.cc:260] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.073246 22616 ts_tablet_manager.cc:1428] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 03:30:10.073236 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.069251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38945 (local address 127.0.0.1:49048)
0929 03:30:10.069301 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.069310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:10.069345 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:10.069751 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:10.069752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:10.069754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:10.069793 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.069795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.071002 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.071003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.071999 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.072001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.072993 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.072994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.073061 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.073078 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 03:30:10.073131 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:30:10.073173 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 03:30:10.073783 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.069432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.6.116.131:38945)
0929 03:30:10.069584 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.069589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.069598 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:10.069632 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.069634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.069647 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.069747 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:10.069872 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.070917 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.072061 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.072898 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.072990 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.073042 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.073671 (+ 629us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.073711 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260929 03:30:10.074497 22928 tablet_service.cc:1467] Processing CreateTablet for tablet 8251887d147043bfb974c1be07cf9af4 (DEFAULT_TABLE table=test-table [id=4d32c94b8a324c2c9457cef60c0e15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:10.074786 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251887d147043bfb974c1be07cf9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:10.075237 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.069356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.0.0.1:48590)
0929 03:30:10.071522 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.071533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:10.071587 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:10.071816 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:10.071816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:10.071818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:10.071912 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.071914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.073122 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.073123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.074165 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.074168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.074989 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.074989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.075033 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.075036 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:10.075166 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 03:30:10.075195 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":75,"threads_started":1}
I20260929 03:30:10.075291 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.069454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.6.116.130:36753)
0929 03:30:10.069666 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.069670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.071654 (+ 1984us) server_negotiation.cc:407] Connection header received
0929 03:30:10.071706 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.071709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.071725 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.071803 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:10.072016 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.073039 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.074237 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.074911 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.074984 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.075032 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.075178 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.075217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":111,"threads_started":1}
I20260929 03:30:10.075847 22796 tablet_service.cc:1467] Processing CreateTablet for tablet 8251887d147043bfb974c1be07cf9af4 (DEFAULT_TABLE table=test-table [id=4d32c94b8a324c2c9457cef60c0e15cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:10.076131 22796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8251887d147043bfb974c1be07cf9af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:10.077569 23004 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Bootstrap starting.
I20260929 03:30:10.078249 23004 tablet_bootstrap.cc:654] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:10.078742 23004 log.cc:824] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:10.078835 23005 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Bootstrap starting.
I20260929 03:30:10.079424 23005 tablet_bootstrap.cc:654] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:10.079559 23004 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: No bootstrap required, opened a new log
I20260929 03:30:10.079612 23004 ts_tablet_manager.cc:1397] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:10.079941 23005 log.cc:824] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:10.080585 23005 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: No bootstrap required, opened a new log
I20260929 03:30:10.080673 23005 ts_tablet_manager.cc:1397] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:10.080819 23004 raft_consensus.cc:348] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.081001 23004 raft_consensus.cc:374] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:10.081051 23004 raft_consensus.cc:729] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a2e3edae5c4d839c33b9ccf07d31d5, State: Initialized, Role: FOLLOWER
I20260929 03:30:10.081177 23004 consensus_queue.cc:260] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.081471 23004 ts_tablet_manager.cc:1428] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:30:10.081493 22994 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
I20260929 03:30:10.082679 23005 raft_consensus.cc:348] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.082820 23005 raft_consensus.cc:374] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:10.082862 23005 raft_consensus.cc:729] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9ea0dcb533458c9b22b515b4f065a6, State: Initialized, Role: FOLLOWER
I20260929 03:30:10.082963 23005 consensus_queue.cc:260] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.083225 23005 ts_tablet_manager.cc:1428] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:10.083276 22862 heartbeater.cc:503] Master 127.6.116.190:36383 was elected leader, sending a full tablet report...
I20260929 03:30:10.135778 23008 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:10.135895 23008 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.136193 23008 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753), 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233)
I20260929 03:30:10.140060 22732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.136434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:51135 (local address 127.6.116.129:38233)
0929 03:30:10.136495 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.136497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.136587 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:30:10.136615 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.136616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.136636 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.136701 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:10.136839 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.137537 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139161 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.139738 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139799 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.139851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.139963 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.140002 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:30:10.140048 22996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.136368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.6.116.131:50753)
0929 03:30:10.136414 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.136425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:10.136462 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:10.136637 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:10.136638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:10.136640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:10.136734 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.136736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.137648 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.137649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.139000 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.139003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.139747 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.139843 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.139847 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:10.139915 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:30:10.139972 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":120}
I20260929 03:30:10.140060 23002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.136459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:50753 (local address 127.6.116.130:36753)
0929 03:30:10.136495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.136497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.136505 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:10.136527 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.136528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.136535 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.136621 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:10.136823 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.137537 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139138 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.139668 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139743 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.139752 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.139963 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.139999 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:30:10.140061 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.136382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:38233 (local address 127.6.116.131:51135)
0929 03:30:10.136516 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.136524 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:10.136571 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:10.136717 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:10.136718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:10.136719 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:10.136768 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.136770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.137648 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.137649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.139000 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:10.139003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.139809 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.139810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:10.139865 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.139868 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:10.139927 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:30:10.139980 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":73,"thread_start_us":71,"threads_started":1}
I20260929 03:30:10.140367 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251887d147043bfb974c1be07cf9af4" candidate_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" is_pre_election: true
I20260929 03:30:10.140360 22678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251887d147043bfb974c1be07cf9af4" candidate_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d744eee3c8746b2959fb242a78de547" is_pre_election: true
I20260929 03:30:10.140486 22816 raft_consensus.cc:2454] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a2e3edae5c4d839c33b9ccf07d31d5 in term 0.
I20260929 03:30:10.140486 22678 raft_consensus.cc:2454] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a2e3edae5c4d839c33b9ccf07d31d5 in term 0.
I20260929 03:30:10.140687 22881 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a2e3edae5c4d839c33b9ccf07d31d5, 8d744eee3c8746b2959fb242a78de547; no voters:
I20260929 03:30:10.140842 23008 raft_consensus.cc:2781] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:10.140893 23008 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:10.140918 23008 raft_consensus.cc:3037] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:10.141592 23008 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.141722 23008 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753), 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233)
I20260929 03:30:10.141846 22816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251887d147043bfb974c1be07cf9af4" candidate_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
I20260929 03:30:10.141844 22678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8251887d147043bfb974c1be07cf9af4" candidate_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d744eee3c8746b2959fb242a78de547"
I20260929 03:30:10.141908 22678 raft_consensus.cc:3037] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:10.141908 22816 raft_consensus.cc:3037] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:10.142680 22816 raft_consensus.cc:2454] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a2e3edae5c4d839c33b9ccf07d31d5 in term 1.
I20260929 03:30:10.142680 22678 raft_consensus.cc:2454] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a2e3edae5c4d839c33b9ccf07d31d5 in term 1.
I20260929 03:30:10.142889 22881 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a2e3edae5c4d839c33b9ccf07d31d5, 8d744eee3c8746b2959fb242a78de547; no voters:
I20260929 03:30:10.143005 23008 raft_consensus.cc:2781] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:10.143168 23008 raft_consensus.cc:686] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 LEADER]: Becoming Leader. State: Replica: 83a2e3edae5c4d839c33b9ccf07d31d5, State: Running, Role: LEADER
I20260929 03:30:10.143271 23008 consensus_queue.cc:237] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:10.143970 22536 catalog_manager.cc:5495] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } health_report { overall_health: UNKNOWN } } }
I20260929 03:30:10.163802 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.161655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.6.116.131:38945)
0929 03:30:10.161690 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:10.161691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:10.161699 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:10.161743 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:10.161744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:10.161756 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:10.161828 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:10.161938 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.162363 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.163018 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:10.163268 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:10.163312 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:10.163446 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:10.163533 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:10.163583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:10.163679 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 03:30:10.163693 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:30:10.165326 22816 raft_consensus.cc:1261] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Refusing update from remote peer 83a2e3edae5c4d839c33b9ccf07d31d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:10.165329 22678 raft_consensus.cc:1261] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Refusing update from remote peer 83a2e3edae5c4d839c33b9ccf07d31d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:10.165632 23011 consensus_queue.cc:1035] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:10.165737 23008 consensus_queue.cc:1035] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:10.167533 23014 mvcc.cc:205] Tried to move back new op lower bound from 7334513091235024896 to 7334513091147481088. Current Snapshot: MvccSnapshot[applied={T|T < 7334513091235024896}]
I20260929 03:30:10.168196 23016 mvcc.cc:205] Tried to move back new op lower bound from 7334513091235024896 to 7334513091147481088. Current Snapshot: MvccSnapshot[applied={T|T < 7334513091235024896}]
I20260929 03:30:10.169517 22534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:10.169600 22534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:10.169711 23015 mvcc.cc:205] Tried to move back new op lower bound from 7334513091235024896 to 7334513091147481088. Current Snapshot: MvccSnapshot[applied={T|T < 7334513091235024896 or (T in {7334513091235024896})}]
I20260929 03:30:10.170275 22534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Sending DeleteTablet for 3 replicas of tablet 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:10.170543 22652 tablet_service.cc:1514] Processing DeleteTablet for tablet 8251887d147043bfb974c1be07cf9af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:10 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:53678
I20260929 03:30:10.170595 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22574
I20260929 03:30:10.170639 22928 tablet_service.cc:1514] Processing DeleteTablet for tablet 8251887d147043bfb974c1be07cf9af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:10 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:49048
I20260929 03:30:10.170809 22733 tablet_replica.cc:330] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: stopping tablet replica
I20260929 03:30:10.170859 22733 raft_consensus.cc:2229] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:10.170907 22733 raft_consensus.cc:2258] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:10.171132 22796 tablet_service.cc:1514] Processing DeleteTablet for tablet 8251887d147043bfb974c1be07cf9af4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:10 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:48590
I20260929 03:30:10.171267 23023 tablet_replica.cc:330] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: stopping tablet replica
I20260929 03:30:10.171353 23023 raft_consensus.cc:2229] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:10.171434 23023 raft_consensus.cc:2258] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:10.171836 23023 ts_tablet_manager.cc:1905] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:10.173105 23023 ts_tablet_manager.cc:1918] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:10.173151 23023 log.cc:1196] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/8251887d147043bfb974c1be07cf9af4
I20260929 03:30:10.173344 23023 ts_tablet_manager.cc:1939] T 8251887d147043bfb974c1be07cf9af4 P 83a2e3edae5c4d839c33b9ccf07d31d5: Deleting consensus metadata
I20260929 03:30:10.173633 22522 catalog_manager.cc:4841] TS 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): tablet 8251887d147043bfb974c1be07cf9af4 (table test-table [id=4d32c94b8a324c2c9457cef60c0e15cc]) successfully deleted
I20260929 03:30:10.175182 23024 tablet_replica.cc:330] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: stopping tablet replica
I20260929 03:30:10.175297 23024 raft_consensus.cc:2229] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:10.175390 23024 raft_consensus.cc:2258] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:10.175717 23024 ts_tablet_manager.cc:1905] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:10.177120 23024 ts_tablet_manager.cc:1918] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:10.177204 23024 log.cc:1196] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/8251887d147043bfb974c1be07cf9af4
W20260929 03:30:10.177451 22520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:38233 (error 108)
I20260929 03:30:10.177526 23024 ts_tablet_manager.cc:1939] T 8251887d147043bfb974c1be07cf9af4 P 1f9ea0dcb533458c9b22b515b4f065a6: Deleting consensus metadata
I20260929 03:30:10.177716 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22729
I20260929 03:30:10.178514 23000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:10.177997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:38233 (local address 127.0.0.1:53694)
0929 03:30:10.178444 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:10.178479 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:38233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
W20260929 03:30:10.178998 22520 catalog_manager.cc:4584] TS 8d744eee3c8746b2959fb242a78de547 (127.6.116.129:38233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8251887d147043bfb974c1be07cf9af4: Network error: Client connection negotiation failed: client connection to 127.6.116.129:38233: connect: Connection refused (error 111)
I20260929 03:30:10.179499 22522 catalog_manager.cc:4841] TS 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): tablet 8251887d147043bfb974c1be07cf9af4 (table test-table [id=4d32c94b8a324c2c9457cef60c0e15cc]) successfully deleted
I20260929 03:30:10.183396 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22865
I20260929 03:30:10.187927 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 22503
I20260929 03:30:30.193070 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--webserver_interface=127.6.116.190
--webserver_port=45625
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:36383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:30.270188 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:30.270404 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:30.270439 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:30.271550 23026 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 03:30:30.271612 23026 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 03:30:30.271634 23026 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 03:30:30.271654 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:30.271682 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:30.271708 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:30.271731 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:30.271759 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:30.273129 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:36383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:36383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:30.273375 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:30.273568 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:30.278422 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.278424 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.278586 23026 server_base.cc:1034] running on GCE node
W20260929 03:30:30.278402 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.278977 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:30.279254 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:30.280398 23026 hybrid_clock.cc:648] HybridClock initialized: now 1790652630280399 us; error 42 us; skew 500 ppm
Sep 29 03:30:30 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:30.286891 23026 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:30:30.287369 23026 webserver.cc:460] Webserver started at http://127.6.116.190:45625/ using document root <none> and password file <none>
I20260929 03:30:30.287575 23026 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:30.287633 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:30.288874 23026 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:30:30.289522 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.289680 23026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:30.289772 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "da505c448b8f4377938a74af24a1bd3c"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "c94d5af729dc85ac8fd8f597507a17d3"
server_key_iv: "717a8b7a3ef6d3c6567fff77711af47d"
server_key_version: "encryptionkey@0"
I20260929 03:30:30.290061 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:30.304719 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:30.306301 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:30.306439 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:30.310284 23026 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:36383
I20260929 03:30:30.310322 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:36383 every 8 connection(s)
I20260929 03:30:30.310670 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:30.311954 23094 sys_catalog.cc:263] Verifying existing consensus state
I20260929 03:30:30.312485 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap starting.
I20260929 03:30:30.317756 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23026
I20260929 03:30:30.318085 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:38233
--local_ip_for_outbound_sockets=127.6.116.129
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=35831
--webserver_interface=127.6.116.129
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:30:30.321000 23094 log.cc:824] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:30.323138 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:30:30.323406 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Bootstrap complete.
I20260929 03:30:30.325500 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:30.325701 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Initialized, Role: FOLLOWER
I20260929 03:30:30.325891 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:30.325965 23094 raft_consensus.cc:388] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:30.326011 23094 raft_consensus.cc:482] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:30.326057 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 2 FOLLOWER]: Advancing to term 3
I20260929 03:30:30.326946 23094 raft_consensus.cc:504] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:30.327025 23094 leader_election.cc:302] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da505c448b8f4377938a74af24a1bd3c; no voters:
I20260929 03:30:30.327240 23094 leader_election.cc:288] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 03:30:30.327275 23098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 03:30:30.327391 23098 raft_consensus.cc:686] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [term 3 LEADER]: Becoming Leader. State: Replica: da505c448b8f4377938a74af24a1bd3c, State: Running, Role: LEADER
I20260929 03:30:30.327484 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:30.327474 23098 consensus_queue.cc:237] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } }
I20260929 03:30:30.327739 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:30:30.327818 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:30.328011 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da505c448b8f4377938a74af24a1bd3c. Latest consensus state: current_term: 3 leader_uuid: "da505c448b8f4377938a74af24a1bd3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da505c448b8f4377938a74af24a1bd3c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 36383 } } }
I20260929 03:30:30.328122 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:30.328186 23102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:30.328966 23102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:30.329093 23102 catalog_manager.cc:1259] Loaded cluster ID: 5a5ec0f491014c7da5393ddaf1e4d99b
I20260929 03:30:30.329321 23102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:30.330567 23102 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:30.330775 23102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Loaded TSK: 0
I20260929 03:30:30.330832 23102 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da505c448b8f4377938a74af24a1bd3c: Loaded TSK: 1
F20260929 03:30:30.331081 23102 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6385b199ad google::LogMessage::Fail() at ??:0
@ 0x7f6385b1c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6385b194de google::LogMessage::Flush() at ??:0
@ 0x7f6385b1aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6388286104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6388283d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f638da771cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f638da7748e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f638da77594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f638da3dc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f638da47ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f63879e70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63879dc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63888406db start_thread at ??:0
@ 0x7f638461761f clone at ??:0
W20260929 03:30:30.414907 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:30.415144 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:30.415172 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:30.416296 23096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:30.416335 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:30.416361 23096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:30.417739 23096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:38233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=35831
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:30.417943 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:30.418181 23096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:30.422912 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.422984 23135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.422914 23136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.423082 23096 server_base.cc:1034] running on GCE node
I20260929 03:30:30.423246 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:30.423496 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:30.424644 23096 hybrid_clock.cc:648] HybridClock initialized: now 1790652630424627 us; error 34 us; skew 500 ppm
Sep 29 03:30:30 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:30.426898 23096 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:30:30.427356 23096 webserver.cc:460] Webserver started at http://127.6.116.129:35831/ using document root <none> and password file <none>
I20260929 03:30:30.427572 23096 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:30.427640 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:30.428892 23096 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:30:30.429584 23145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.429757 23096 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:30:30.429860 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "8d744eee3c8746b2959fb242a78de547"
format_stamp: "Formatted at 2026-09-29 03:29:48 on dist-test-slave-z2zg"
server_key: "cdbf43ef63d43697218c25e9c5b6f562"
server_key_iv: "76e5bc3c0aaebd45615f2d3ddfe6ff51"
server_key_version: "encryptionkey@0"
I20260929 03:30:30.430152 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:30.442072 23096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:30.444144 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:30.444289 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:30.444496 23096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:30.444862 23152 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 03:30:30.446579 23096 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 03:30:30.446637 23096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 03:30:30.446689 23096 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 03:30:30.447312 23096 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 03:30:30.447362 23096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:30.447419 23152 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Bootstrap starting.
I20260929 03:30:30.454279 23096 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:38233
I20260929 03:30:30.454398 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:38233 every 8 connection(s)
I20260929 03:30:30.454675 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:30.458144 23152 log.cc:824] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:30.460553 23152 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 03:30:30.460877 23152 tablet_bootstrap.cc:492] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Bootstrap complete.
I20260929 03:30:30.462100 23152 ts_tablet_manager.cc:1397] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260929 03:30:30.463294 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23096
I20260929 03:30:30.463541 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:36753
--local_ip_for_outbound_sockets=127.6.116.130
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=35985
--webserver_interface=127.6.116.130
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:30:30.463528 23152 raft_consensus.cc:348] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:30.463847 23152 raft_consensus.cc:729] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d744eee3c8746b2959fb242a78de547, State: Initialized, Role: FOLLOWER
I20260929 03:30:30.463989 23152 consensus_queue.cc:260] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:30.464238 23152 ts_tablet_manager.cc:1428] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260929 03:30:30.519497 23262 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:30:30.455305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:40509)
0929 03:30:30.455454 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.455491 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:30.455910 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:30.519301 (+ 63391us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
W20260929 03:30:30.519743 23260 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:36383 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:36383: Client connection negotiation failed: client connection to 127.6.116.190:36383: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 03:30:30.520056 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.519943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:34537)
0929 03:30:30.519993 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.520031 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 03:30:30.520399 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.520307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:42633)
0929 03:30:30.520356 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.520381 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:30:30.520516 23260 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 03:30:30.541849 23265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:30.542023 23265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:30.542045 23265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:30.543215 23265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:30.543253 23265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:30.543298 23265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:30.544689 23265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=35985
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:30.544906 23265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:30.545115 23265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:30.549645 23270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.549661 23273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.549646 23271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.549860 23265 server_base.cc:1034] running on GCE node
I20260929 03:30:30.550115 23265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:30.550350 23265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:30.551492 23265 hybrid_clock.cc:648] HybridClock initialized: now 1790652630551481 us; error 23 us; skew 500 ppm
Sep 29 03:30:30 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:30.553642 23265 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:30:30.554072 23265 webserver.cc:460] Webserver started at http://127.6.116.130:35985/ using document root <none> and password file <none>
I20260929 03:30:30.554293 23265 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:30.554356 23265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:30.555680 23265 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:30.556177 23280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.556334 23265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:30.556438 23265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "1f9ea0dcb533458c9b22b515b4f065a6"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "6e196f8ef52e744f3802d5e65dd688f4"
server_key_iv: "d27d774f593bbcfbf90e4d30a7aad8b5"
server_key_version: "encryptionkey@0"
I20260929 03:30:30.556727 23265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:30.570986 23265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:30.572546 23265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:30.572714 23265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:30.572911 23265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:30.573175 23265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:30.573223 23265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.573256 23265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:30.573282 23265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.578975 23265 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:36753
I20260929 03:30:30.579025 23393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:36753 every 8 connection(s)
I20260929 03:30:30.579334 23265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:30.580367 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.580022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:34143)
0929 03:30:30.580162 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.580248 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260929 03:30:30.580720 23394 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:36383 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:36383: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
I20260929 03:30:30.580953 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.580876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:33591)
0929 03:30:30.580914 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.580938 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:30:30.581212 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.581138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:36297)
0929 03:30:30.581171 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.581200 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 03:30:30.581310 23394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:30:30.588581 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23265
I20260929 03:30:30.588792 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:38945
--local_ip_for_outbound_sockets=127.6.116.131
--tserver_master_addrs=127.6.116.190:36383
--webserver_port=38941
--webserver_interface=127.6.116.131
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:30.666098 23397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:30.666265 23397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:30.666287 23397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:30.667304 23397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:30.667335 23397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:30.667357 23397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:30.668630 23397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:38945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=38941
--tserver_master_addrs=127.6.116.190:36383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:30.668781 23397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:30.668926 23397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:30.673591 23405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.673681 23397 server_base.cc:1034] running on GCE node
W20260929 03:30:30.673614 23403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:30.673805 23402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:30.674024 23397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:30.674311 23397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:30.675472 23397 hybrid_clock.cc:648] HybridClock initialized: now 1790652630675457 us; error 25 us; skew 500 ppm
Sep 29 03:30:30 dist-test-slave-z2zg krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:30.677691 23397 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:30:30.678092 23397 webserver.cc:460] Webserver started at http://127.6.116.131:38941/ using document root <none> and password file <none>
I20260929 03:30:30.678261 23397 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:30.678299 23397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:30.679539 23397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:30.680042 23412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.680233 23397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.680292 23397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "83a2e3edae5c4d839c33b9ccf07d31d5"
format_stamp: "Formatted at 2026-09-29 03:29:49 on dist-test-slave-z2zg"
server_key: "1a3bd670ea55f5285500ab3811e8e1f3"
server_key_iv: "d1c24cdbe0f241f63877c8b78a39085c"
server_key_version: "encryptionkey@0"
I20260929 03:30:30.680545 23397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:30.693990 23397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:30.695605 23397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:30.695703 23397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:30.695848 23397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:30.696058 23397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:30.696087 23397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.696105 23397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:30.696117 23397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:30.701805 23397 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:38945
I20260929 03:30:30.701849 23525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:38945 every 8 connection(s)
I20260929 03:30:30.702137 23397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:30:30.702967 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.702656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:37327)
0929 03:30:30.702793 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.702850 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260929 03:30:30.703253 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23397
W20260929 03:30:30.703326 23526 heartbeater.cc:644] Failed to heartbeat to 127.6.116.190:36383 (0 consecutive failures): Network error: Failed to ping master at 127.6.116.190:36383: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
I20260929 03:30:30.703610 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.703517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:50865)
0929 03:30:30.703565 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.703589 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:30:30.703913 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:30.703835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:51283)
0929 03:30:30.703873 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:30.703894 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:30:30.704042 23526 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 03:30:31.521073 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:31.520881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:59823)
0929 03:30:31.520988 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:31.521019 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260929 03:30:31.581781 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:31.581593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:45039)
0929 03:30:31.581714 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:31.581741 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260929 03:30:31.704461 23532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:31.704299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:45915)
0929 03:30:31.704390 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:31.704418 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
I20260929 03:30:32.237573 23533 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:32.237679 23533 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:32.238003 23533 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945), 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
Sep 29 03:30:32 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.130@KRBTEST.COM
Sep 29 03:30:32 dist-test-slave-z2zg krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.131@KRBTEST.COM
I20260929 03:30:32.248195 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.238449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:45529 (local address 127.6.116.130:36753)
0929 03:30:32.238560 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:32.238570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:32.238589 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:30:32.238989 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:32.238990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:32.239025 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:32.239217 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:32.240005 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.240894 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.241798 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.242183 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.242300 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:32.244677 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:32.244680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:32.244680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:32.244689 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:32.246537 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:32.246936 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:32.246938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:32.246939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:32.247014 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:32.247290 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:32.247291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:32.247291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:32.247593 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:32.247683 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:32.247960 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 03:30:32.248085 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":63,"threads_started":1}
I20260929 03:30:32.248204 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.238273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:35819 (local address 127.6.116.131:38945)
0929 03:30:32.238485 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:32.238499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:32.238538 (+ 39us) server_negotiation.cc:407] Connection header received
0929 03:30:32.238980 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:32.238982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:32.239024 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:32.239220 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:32.240017 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.240931 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.241802 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.242228 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.242353 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:32.245049 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:32.245052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:32.245052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:32.245061 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:32.246769 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:32.247148 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:32.247154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:32.247155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:32.247240 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:32.247438 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:32.247438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:32.247439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:32.247706 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:32.247768 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:32.247959 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 03:30:32.248085 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":54,"threads_started":1}
I20260929 03:30:32.248206 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.238171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:38945 (local address 127.6.116.129:35819)
0929 03:30:32.238265 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:32.238274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:32.238349 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:32.239289 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:32.239290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:32.239697 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:32.239875 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:32.239881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.241044 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.241045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:32.241709 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:32.241711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.242322 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.242324 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:32.242397 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:32.242936 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:32.242945 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:32.244962 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:32.246840 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:32.246841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:32.246841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:32.247013 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:32.247323 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:32.247324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:32.247324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:32.247369 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:32.247770 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:32.247771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:32.247800 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 03:30:32.247895 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:30:32.248093 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":74,"thread_start_us":51,"threads_started":1}
I20260929 03:30:32.248672 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.238333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:36753 (local address 127.6.116.129:45529)
0929 03:30:32.238420 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:32.238432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:32.238489 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:32.239268 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:32.239271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:32.239694 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:32.239875 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:32.239881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.240988 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.240989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:32.241707 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:32.241709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:32.242276 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:32.242277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:32.242383 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:32.242997 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:32.243002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:32.244581 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:32.246614 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:32.246620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:32.246621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:32.246852 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:32.247124 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:32.247125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:32.247125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:32.247226 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:32.247660 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:32.247665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:32.247787 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 03:30:32.247895 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 03:30:32.248146 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":101,"thread_start_us":46,"threads_started":1}
W20260929 03:30:32.249105 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
W20260929 03:30:32.249260 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:32.249311 23148 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d744eee3c8746b2959fb242a78de547; no voters: 1f9ea0dcb533458c9b22b515b4f065a6, 83a2e3edae5c4d839c33b9ccf07d31d5
I20260929 03:30:32.249405 23533 raft_consensus.cc:2726] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:30:32.521605 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.521489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:34435)
0929 03:30:32.521537 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:32.521568 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:30:32.582290 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.582097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:50701)
0929 03:30:32.582213 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:32.582252 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260929 03:30:32.705034 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:32.704806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:42651)
0929 03:30:32.704953 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:32.704995 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":53,"threads_started":1}
I20260929 03:30:33.522389 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:33.522090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:38119)
0929 03:30:33.522306 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:33.522343 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":55,"threads_started":1}
I20260929 03:30:33.583087 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:33.582818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:55431)
0929 03:30:33.582983 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:33.583017 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260929 03:30:33.705591 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:33.705434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:50297)
0929 03:30:33.705532 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:33.705559 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260929 03:30:34.434881 23543 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:34.434963 23543 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:34.435122 23543 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945), 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
W20260929 03:30:34.435475 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
W20260929 03:30:34.435545 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:34.435563 23148 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d744eee3c8746b2959fb242a78de547; no voters: 1f9ea0dcb533458c9b22b515b4f065a6, 83a2e3edae5c4d839c33b9ccf07d31d5
I20260929 03:30:34.435643 23543 raft_consensus.cc:2726] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:30:34.523021 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:34.522818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:42961)
0929 03:30:34.522942 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:34.522972 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260929 03:30:34.583709 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:34.583494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:48861)
0929 03:30:34.583624 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:34.583659 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260929 03:30:34.706120 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:34.705941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:51487)
0929 03:30:34.706051 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:34.706084 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260929 03:30:35.523682 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:35.523474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:33415)
0929 03:30:35.523605 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:35.523639 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260929 03:30:35.584223 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:35.584051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:42515)
0929 03:30:35.584155 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:35.584189 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":51,"threads_started":1}
I20260929 03:30:35.706676 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:35.706503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:47397)
0929 03:30:35.706604 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:35.706633 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260929 03:30:36.524326 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:36.524126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:52991)
0929 03:30:36.524251 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:36.524281 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260929 03:30:36.584811 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:36.584611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:36441)
0929 03:30:36.584726 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:36.584771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260929 03:30:36.707265 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:36.707092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:35103)
0929 03:30:36.707198 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:36.707230 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
I20260929 03:30:37.524897 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:37.524716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:42721)
0929 03:30:37.524824 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:37.524856 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260929 03:30:37.585489 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:37.585179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:50963)
0929 03:30:37.585418 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:37.585454 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":52,"threads_started":1}
I20260929 03:30:37.693104 23555 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:37.693207 23555 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:37.693377 23555 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945), 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
W20260929 03:30:37.693765 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
W20260929 03:30:37.693840 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:37.693856 23148 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d744eee3c8746b2959fb242a78de547; no voters: 1f9ea0dcb533458c9b22b515b4f065a6, 83a2e3edae5c4d839c33b9ccf07d31d5
I20260929 03:30:37.693959 23555 raft_consensus.cc:2726] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:30:37.707818 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:37.707637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:48437)
0929 03:30:37.707754 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:37.707786 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260929 03:30:38.525689 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:38.525400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:59839)
0929 03:30:38.525611 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:38.525646 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":53,"threads_started":1}
I20260929 03:30:38.586114 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:38.585927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:53445)
0929 03:30:38.586045 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:38.586079 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260929 03:30:38.708621 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:38.708357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:53525)
0929 03:30:38.708550 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:38.708585 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":64,"threads_started":1}
I20260929 03:30:39.526464 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:39.526199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:46575)
0929 03:30:39.526366 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:39.526413 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260929 03:30:39.586724 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:39.586510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:32989)
0929 03:30:39.586653 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:39.586688 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260929 03:30:39.709360 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:39.709114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:52429)
0929 03:30:39.709287 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:39.709323 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260929 03:30:40.527127 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:40.526884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:51611)
0929 03:30:40.527012 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:40.527086 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260929 03:30:40.587328 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:40.587145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:58323)
0929 03:30:40.587259 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:40.587291 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260929 03:30:40.710071 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:40.709887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:45617)
0929 03:30:40.710005 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:40.710036 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260929 03:30:41.527736 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:41.527536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:43749)
0929 03:30:41.527652 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:41.527690 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260929 03:30:41.587874 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:41.587685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:33479)
0929 03:30:41.587805 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:41.587838 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260929 03:30:41.614951 23568 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:41.615026 23568 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:41.615242 23568 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945), 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
W20260929 03:30:41.615597 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
W20260929 03:30:41.615684 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:41.615715 23148 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d744eee3c8746b2959fb242a78de547; no voters: 1f9ea0dcb533458c9b22b515b4f065a6, 83a2e3edae5c4d839c33b9ccf07d31d5
I20260929 03:30:41.615806 23568 raft_consensus.cc:2726] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:30:41.710680 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:41.710482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:49807)
0929 03:30:41.710592 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:41.710624 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
I20260929 03:30:42.528345 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:42.528152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:37937)
0929 03:30:42.528266 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:42.528300 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260929 03:30:42.588637 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:42.588414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:59673)
0929 03:30:42.588531 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:42.588584 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260929 03:30:42.711397 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:42.711168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:34753)
0929 03:30:42.711323 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:42.711360 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260929 03:30:43.528965 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:43.528755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:51701)
0929 03:30:43.528881 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:43.528918 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260929 03:30:43.589353 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:43.589111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:48289)
0929 03:30:43.589267 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:43.589312 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260929 03:30:43.711959 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:43.711763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:46825)
0929 03:30:43.711890 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:43.711921 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260929 03:30:44.529706 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:44.529498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:47225)
0929 03:30:44.529619 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:44.529660 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260929 03:30:44.590009 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:44.589745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:38739)
0929 03:30:44.589940 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:44.589974 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":64,"threads_started":1}
I20260929 03:30:44.712555 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:44.712372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:32897)
0929 03:30:44.712485 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:44.712517 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260929 03:30:45.530435 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:45.530234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.129:54111)
0929 03:30:45.530356 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:45.530390 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260929 03:30:45.590755 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:45.590473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.130:36305)
0929 03:30:45.590685 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:45.590719 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":52,"threads_started":1}
I20260929 03:30:45.633271 23581 raft_consensus.cc:482] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:45.633368 23581 raft_consensus.cc:504] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a2e3edae5c4d839c33b9ccf07d31d5" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 38945 } } peers { permanent_uuid: "1f9ea0dcb533458c9b22b515b4f065a6" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 36753 } } peers { permanent_uuid: "8d744eee3c8746b2959fb242a78de547" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 38233 } }
I20260929 03:30:45.633591 23581 leader_election.cc:288] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945), 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753)
W20260929 03:30:45.634021 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83a2e3edae5c4d839c33b9ccf07d31d5 (127.6.116.131:38945): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
W20260929 03:30:45.634121 23148 leader_election.cc:341] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1f9ea0dcb533458c9b22b515b4f065a6 (127.6.116.130:36753): Not found: Tablet not found: 8251887d147043bfb974c1be07cf9af4
I20260929 03:30:45.634157 23148 leader_election.cc:302] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d744eee3c8746b2959fb242a78de547; no voters: 1f9ea0dcb533458c9b22b515b4f065a6, 83a2e3edae5c4d839c33b9ccf07d31d5
I20260929 03:30:45.634254 23581 raft_consensus.cc:2726] T 8251887d147043bfb974c1be07cf9af4 P 8d744eee3c8746b2959fb242a78de547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 03:30:45.708397 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23096
I20260929 03:30:45.713178 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23265
I20260929 03:30:45.713229 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:45.713009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:36383 (local address 127.6.116.131:46901)
0929 03:30:45.713151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:45.713183 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.190:36383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260929 03:30:45.717574 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23397
2026-09-29T03:30:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58250 ms)
[----------] 30 tests from SecurityITest (225160 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:45 dist-test-slave-z2zg krb5kdc[23585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:45 dist-test-slave-z2zg krb5kdc[23585](info): set up 2 sockets
Sep 29 03:30:45 dist-test-slave-z2zg krb5kdc[23585](info): commencing operation
krb5kdc: starting...
W20260929 03:30:46.740633 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:46 dist-test-slave-z2zg krb5kdc[23585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:30:46.849259 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40717
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:46.926192 23598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:46.926395 23598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:46.926429 23598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:46.927603 23598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:46.927659 23598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:46.927681 23598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:46.927707 23598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:46.927737 23598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:46.929134 23598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:40717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:46.929373 23598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:46.929579 23598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:30:46.934429 23598 server_base.cc:1034] running on GCE node
W20260929 03:30:46.934374 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:46.934398 23606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:46.934405 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:46.934891 23598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:46.935207 23598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:46.936347 23598 hybrid_clock.cc:648] HybridClock initialized: now 1790652646936333 us; error 28 us; skew 500 ppm
Sep 29 03:30:46 dist-test-slave-z2zg krb5kdc[23585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652646, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:46.938634 23598 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:30:46.939033 23598 webserver.cc:460] Webserver started at http://127.6.116.190:43943/ using document root <none> and password file <none>
I20260929 03:30:46.939316 23598 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:46.939361 23598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:46.939512 23598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:46.940459 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "a66d80c176a84e78822b7a05fb1ec8ab"
format_stamp: "Formatted at 2026-09-29 03:30:46 on dist-test-slave-z2zg"
server_key: "3374a2fd785d59905e2d1dd1e145bca8"
server_key_iv: "f3f98e8225813ab3386c716cd353a73a"
server_key_version: "encryptionkey@0"
I20260929 03:30:46.940798 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "a66d80c176a84e78822b7a05fb1ec8ab"
format_stamp: "Formatted at 2026-09-29 03:30:46 on dist-test-slave-z2zg"
server_key: "3374a2fd785d59905e2d1dd1e145bca8"
server_key_iv: "f3f98e8225813ab3386c716cd353a73a"
server_key_version: "encryptionkey@0"
I20260929 03:30:46.942025 23598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:46.942667 23613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:46.942839 23598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:46.942937 23598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "a66d80c176a84e78822b7a05fb1ec8ab"
format_stamp: "Formatted at 2026-09-29 03:30:46 on dist-test-slave-z2zg"
server_key: "3374a2fd785d59905e2d1dd1e145bca8"
server_key_iv: "f3f98e8225813ab3386c716cd353a73a"
server_key_version: "encryptionkey@0"
I20260929 03:30:46.943009 23598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:46.954458 23598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:46.956073 23598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:46.956231 23598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:46.959766 23598 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:40717
I20260929 03:30:46.959795 23665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:40717 every 8 connection(s)
I20260929 03:30:46.960091 23598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:46.960889 23666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:46.962780 23666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: Bootstrap starting.
I20260929 03:30:46.963212 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23598
I20260929 03:30:46.963318 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:46.963387 23666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:46.963542 6610 external_mini_cluster.cc:1428] Setting key 195e88d7527773ba740737fbcb6f9682
I20260929 03:30:46.963742 23666 log.cc:824] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:46.964444 23666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: No bootstrap required, opened a new log
I20260929 03:30:46.966691 23666 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } }
I20260929 03:30:46.966840 23666 raft_consensus.cc:374] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:46.966873 23666 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66d80c176a84e78822b7a05fb1ec8ab, State: Initialized, Role: FOLLOWER
I20260929 03:30:46.966987 23666 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } }
I20260929 03:30:46.967069 23666 raft_consensus.cc:388] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:46.967098 23666 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:46.967151 23666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:46.967820 23666 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } }
I20260929 03:30:46.967954 23666 leader_election.cc:302] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a66d80c176a84e78822b7a05fb1ec8ab; no voters:
I20260929 03:30:46.968127 23666 leader_election.cc:288] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:46.968210 23671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:46.968360 23666 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:46.968391 23671 raft_consensus.cc:686] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [term 1 LEADER]: Becoming Leader. State: Replica: a66d80c176a84e78822b7a05fb1ec8ab, State: Running, Role: LEADER
I20260929 03:30:46.968531 23671 consensus_queue.cc:237] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } }
I20260929 03:30:46.968928 23672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } } }
I20260929 03:30:46.969074 23672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:46.969497 23673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66d80c176a84e78822b7a05fb1ec8ab. Latest consensus state: current_term: 1 leader_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66d80c176a84e78822b7a05fb1ec8ab" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 40717 } } }
I20260929 03:30:46.969617 23673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:46.969790 23682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 29 03:30:46 dist-test-slave-z2zg krb5kdc[23585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:46.970292 23682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:46.972198 23682 catalog_manager.cc:1347] Generated new cluster ID: b9461b1bf52244eaa79a3d9b836cf506
I20260929 03:30:46.972254 23682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:46.974843 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:46.963994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.6.116.190:40717)
0929 03:30:46.964156 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:46.964165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:46.964197 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:30:46.964638 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:46.964644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:46.964686 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:46.964818 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:46.965588 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:46.966425 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:46.967454 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:46.967830 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:46.967938 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:46.970893 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:46.970897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:46.970898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:46.970909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:46.972992 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:46.973357 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:46.973360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:46.973360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:46.973446 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:46.973608 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:46.973609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:46.973609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:46.973847 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:46.974346 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:46.974414 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:30:46.974525 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260929 03:30:46.987015 23682 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:46.987463 23682 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:46.992798 23682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: Generated new TSK 0
I20260929 03:30:46.992964 23682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:30:47.022044 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:47.100282 23692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.100471 23692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.100495 23692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.101537 23692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.101569 23692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.101595 23692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:47.102916 23692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.103147 23692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.103322 23692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.107856 23697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.107877 23698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.107856 23700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.108121 23692 server_base.cc:1034] running on GCE node
I20260929 03:30:47.108292 23692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.108510 23692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.109664 23692 hybrid_clock.cc:648] HybridClock initialized: now 1790652647109648 us; error 28 us; skew 500 ppm
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:47.112177 23692 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:30:47.112584 23692 webserver.cc:460] Webserver started at http://127.6.116.129:35065/ using document root <none> and password file <none>
I20260929 03:30:47.112752 23692 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.112790 23692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.112870 23692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.113755 23692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "8248f86c05064949993d549fa8f51e4a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "ad8f9f20ecf284d96436f7649b16ca4d"
server_key_iv: "fe2b3064214ff3023e6c83b7a34d788f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.114054 23692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "8248f86c05064949993d549fa8f51e4a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "ad8f9f20ecf284d96436f7649b16ca4d"
server_key_iv: "fe2b3064214ff3023e6c83b7a34d788f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.115291 23692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:47.115950 23707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.116139 23692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:47.116215 23692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "8248f86c05064949993d549fa8f51e4a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "ad8f9f20ecf284d96436f7649b16ca4d"
server_key_iv: "fe2b3064214ff3023e6c83b7a34d788f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.116266 23692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:47.127665 23692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:47.129246 23692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:47.129343 23692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:47.129495 23692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:47.129911 23692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:47.129951 23692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.129971 23692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:47.129983 23692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.135619 23692 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:34119
I20260929 03:30:47.135691 23820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:34119 every 8 connection(s)
I20260929 03:30:47.135980 23692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:47.136198 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23692
I20260929 03:30:47.136288 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:47.136497 6610 external_mini_cluster.cc:1428] Setting key 87a5b50ac6d8aef34e1cdd4eb13ce067
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:30:47.145834 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.136656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:35959 (local address 127.6.116.190:40717)
0929 03:30:47.136703 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.136705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.136901 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:30:47.137433 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.137434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.137446 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.137490 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.138437 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.139202 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.140123 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.140480 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.140558 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.142596 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.142598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.142598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:47.142612 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.144296 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.144727 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.144729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.144729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.144789 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.145126 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.145127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.145128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.145240 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.145318 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.145633 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.145744 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:30:47.145982 23823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.136630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40717 (local address 127.6.116.129:35959)
0929 03:30:47.136793 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.136833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.137328 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.137545 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.137549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.138105 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:47.138321 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.138327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.139283 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.139284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.140029 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.140031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.140583 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.140583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.140684 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.141213 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:47.141220 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:47.142505 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:47.144391 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.144397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.144399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.144610 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.144893 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.144894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.144895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.144999 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.145346 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:47.145352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:47.145486 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.145612 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.145772 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260929 03:30:47.146495 23821 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40717
I20260929 03:30:47.146602 23821 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:47.146770 23821 heartbeater.cc:511] Master 127.6.116.190:40717 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
I20260929 03:30:47.147330 23630 ts_manager.cc:194] Registered new tserver with Master: 8248f86c05064949993d549fa8f51e4a (127.6.116.129:34119)
I20260929 03:30:47.148249 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:35959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:30:47.160028 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:47.238533 23826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.238761 23826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.238798 23826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.239933 23826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.240002 23826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.240054 23826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:47.241467 23826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.241721 23826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.241953 23826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.246418 23834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.246412 23832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.246579 23826 server_base.cc:1034] running on GCE node
W20260929 03:30:47.246439 23831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.246840 23826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.247110 23826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.248255 23826 hybrid_clock.cc:648] HybridClock initialized: now 1790652647248246 us; error 39 us; skew 500 ppm
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:47.250473 23826 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:30:47.250865 23826 webserver.cc:460] Webserver started at http://127.6.116.130:39237/ using document root <none> and password file <none>
I20260929 03:30:47.251119 23826 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.251185 23826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.251312 23826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.252215 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "5e3b5a8a8d0b43908e833f89161d730a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "23539171b07e543c195fc6843c2cff4b"
server_key_iv: "1f1e57dbd646552677d1bb6cd54fdf94"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.252558 23826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "5e3b5a8a8d0b43908e833f89161d730a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "23539171b07e543c195fc6843c2cff4b"
server_key_iv: "1f1e57dbd646552677d1bb6cd54fdf94"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.253811 23826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:47.254462 23841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.254624 23826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:47.254709 23826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "5e3b5a8a8d0b43908e833f89161d730a"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "23539171b07e543c195fc6843c2cff4b"
server_key_iv: "1f1e57dbd646552677d1bb6cd54fdf94"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.254783 23826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:47.263775 23826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:47.265391 23826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:47.265529 23826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:47.265718 23826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:47.266007 23826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:47.266062 23826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.266098 23826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:47.266125 23826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.271972 23826 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45323
I20260929 03:30:47.272017 23954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45323 every 8 connection(s)
I20260929 03:30:47.272310 23826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:47.273833 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23826
I20260929 03:30:47.273904 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:47.274087 6610 external_mini_cluster.cc:1428] Setting key 0979bb5b9a547e163375ecae1606d561
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:47.281895 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.272947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:60259 (local address 127.6.116.190:40717)
0929 03:30:47.273004 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.273006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.273168 (+ 162us) server_negotiation.cc:407] Connection header received
0929 03:30:47.273542 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.273543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.273552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.273601 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.274556 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.275187 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.276529 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.276851 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.276924 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.279222 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.279224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.279224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:47.279230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.280515 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.280987 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.280988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.280988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.281035 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.281333 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.281333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.281334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.281425 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.281475 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.281770 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.281830 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:30:47.282140 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.272921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40717 (local address 127.6.116.130:60259)
0929 03:30:47.273053 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.273088 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.273467 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.273643 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.273646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.274241 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:47.274466 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.274473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.275269 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.275270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.276446 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.276449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.276954 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.276955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.277053 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.277515 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:47.277522 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:47.279137 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:47.280598 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.280604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.280606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.280913 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.281110 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.281111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.281111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.281254 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.281490 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:47.281495 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:47.281643 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.281753 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.281907 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:30:47.282604 23955 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40717
I20260929 03:30:47.282703 23955 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:47.282855 23955 heartbeater.cc:511] Master 127.6.116.190:40717 requested a full tablet report, sending...
I20260929 03:30:47.283209 23630 ts_manager.cc:194] Registered new tserver with Master: 5e3b5a8a8d0b43908e833f89161d730a (127.6.116.130:45323)
I20260929 03:30:47.283514 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:60259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:30:47.298465 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:41675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:47.376204 23960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.376379 23960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.376402 23960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.377406 23960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.377439 23960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.377463 23960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:47.378723 23960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:41675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:40717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.378882 23960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.379025 23960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.383905 23965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.383884 23968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.383884 23966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.383912 23960 server_base.cc:1034] running on GCE node
I20260929 03:30:47.384310 23960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.384485 23960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.385638 23960 hybrid_clock.cc:648] HybridClock initialized: now 1790652647385623 us; error 26 us; skew 500 ppm
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:47.388141 23960 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:30:47.388521 23960 webserver.cc:460] Webserver started at http://127.6.116.131:33221/ using document root <none> and password file <none>
I20260929 03:30:47.388747 23960 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.388824 23960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.388965 23960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.389928 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "d32d4c2a2ed448958257d732aee6b192"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "0b1716c057eb6ab18f4158294d53c4a4"
server_key_iv: "a16e6d5324b0040e769b8c1a036bfb7f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.390270 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "d32d4c2a2ed448958257d732aee6b192"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "0b1716c057eb6ab18f4158294d53c4a4"
server_key_iv: "a16e6d5324b0040e769b8c1a036bfb7f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.391577 23960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:30:47.392279 23975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.392438 23960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:47.392530 23960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "d32d4c2a2ed448958257d732aee6b192"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "0b1716c057eb6ab18f4158294d53c4a4"
server_key_iv: "a16e6d5324b0040e769b8c1a036bfb7f"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.392606 23960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:47.406366 23960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:47.408125 23960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:47.408273 23960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:47.408465 23960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:47.408735 23960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:47.408802 23960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.408840 23960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:47.408869 23960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.414289 23960 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:34115
I20260929 03:30:47.414350 24088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:34115 every 8 connection(s)
I20260929 03:30:47.414636 23960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:30:47 dist-test-slave-z2zg krb5kdc[23585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652647, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:47.422221 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 23960
I20260929 03:30:47.422345 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:47.422538 6610 external_mini_cluster.cc:1428] Setting key 213d3cea7dc1409ba56b72036779ee8e
I20260929 03:30:47.422983 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.415351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40115 (local address 127.6.116.190:40717)
0929 03:30:47.415404 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.415406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.415601 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:30:47.416009 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.416009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.416018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.416064 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.416851 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.417326 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.418184 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.418373 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.418422 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.420368 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.420370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.420370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:47.420378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.421520 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.421924 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.421925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.421925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.421977 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.422251 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.422252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.422252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.422423 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.422490 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.422791 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.422887 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:30:47.423170 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.415334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:40717 (local address 127.6.116.131:40115)
0929 03:30:47.415491 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.415528 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.415935 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.416116 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.416119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.416610 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:47.416782 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.416786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.417399 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.417399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.418111 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.418112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.418440 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.418441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.418533 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.418974 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:47.418983 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:47.420276 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:47.421612 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.421618 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.421620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.421828 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.422047 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:47.422047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:47.422048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:47.422178 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:47.422484 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:47.422491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:47.422650 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.422751 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.422926 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260929 03:30:47.423528 24089 heartbeater.cc:347] Connected to a master server at 127.6.116.190:40717
I20260929 03:30:47.423635 24089 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:47.423803 24089 heartbeater.cc:511] Master 127.6.116.190:40717 requested a full tablet report, sending...
I20260929 03:30:47.424117 23630 ts_manager.cc:194] Registered new tserver with Master: d32d4c2a2ed448958257d732aee6b192 (127.6.116.131:34115)
I20260929 03:30:47.424458 23630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:40115
I20260929 03:30:47.434551 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:47.441072 23670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.435808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.6.116.190:40717)
0929 03:30:47.435853 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.435854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.435980 (+ 126us) server_negotiation.cc:407] Connection header received
0929 03:30:47.436050 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.436051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.436059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.436108 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.436674 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.437104 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.437846 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.438031 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.438107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.439164 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.439165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.439166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:47.439171 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.440236 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.440513 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.440513 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.440513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.440556 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:47.440693 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:47.440694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:47.440694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:47.440820 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.440890 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.440965 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.441019 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:30:47.442408 23630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:47.444173 23630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:47.449611 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45323 (local address 127.0.0.1:55480)
0929 03:30:47.446354 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.446385 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.446475 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.446776 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.446779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.446781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:47.446852 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.446854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.447773 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.447774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.448517 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.448519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.449342 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.449343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.449420 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.449431 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.449499 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.449546 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":5,"thread_start_us":68,"threads_started":1}
I20260929 03:30:47.449628 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.6.116.130:45323)
0929 03:30:47.446329 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.446334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.446458 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:30:47.446560 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.446563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.446577 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.446693 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:47.446932 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.447660 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.448621 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.449255 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.449342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.449394 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.449516 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.449555 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":100,"threads_started":1}
I20260929 03:30:47.450371 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.6.116.131:34115)
0929 03:30:47.446532 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.446536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.446665 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:30:47.446703 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.446706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.446719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.446792 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:47.447011 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.448034 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.449447 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.450086 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.450153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.450202 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.450299 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.450324 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260929 03:30:47.450387 24098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:34115 (local address 127.0.0.1:60426)
0929 03:30:47.446575 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.446583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.446651 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.446902 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.446903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.446905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:47.446935 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.446936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.448147 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.448148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.449356 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.449359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.450168 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.450169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.450226 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.450230 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.450290 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.450326 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":76,"thread_start_us":79,"threads_started":1}
I20260929 03:30:47.450595 23889 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8df219857340aab280f914bddfb14a (DEFAULT_TABLE table=test-table [id=e8a0200d17d44e86ba4fc4387f4be4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:47.450925 23889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8df219857340aab280f914bddfb14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:47.451205 24023 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8df219857340aab280f914bddfb14a (DEFAULT_TABLE table=test-table [id=e8a0200d17d44e86ba4fc4387f4be4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:47.451475 24023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8df219857340aab280f914bddfb14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:47.454228 24103 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Bootstrap starting.
I20260929 03:30:47.454536 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34119 (local address 127.0.0.1:52212)
0929 03:30:47.446403 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.446415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.446500 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.446780 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.446781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.446783 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:47.446844 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.446846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.448156 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.448157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.449356 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.449359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.454303 (+ 4944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.454304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.454369 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.454374 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.454439 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.454477 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":73,"thread_start_us":71,"threads_started":1}
I20260929 03:30:47.454581 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.446207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52212 (local address 127.6.116.129:34119)
0929 03:30:47.446348 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.446352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.446470 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:30:47.446599 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.446602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.446617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.446672 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:47.446928 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.448034 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.449464 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.450111 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.454201 (+ 4090us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.454266 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.454462 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.454502 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260929 03:30:47.454841 24104 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Bootstrap starting.
I20260929 03:30:47.454989 24103 tablet_bootstrap.cc:654] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:47.455281 23755 tablet_service.cc:1467] Processing CreateTablet for tablet 8b8df219857340aab280f914bddfb14a (DEFAULT_TABLE table=test-table [id=e8a0200d17d44e86ba4fc4387f4be4c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:47.455521 24103 log.cc:824] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:47.455554 23755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b8df219857340aab280f914bddfb14a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:47.456193 24103 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: No bootstrap required, opened a new log
I20260929 03:30:47.456243 24103 ts_tablet_manager.cc:1397] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:47.457803 24103 raft_consensus.cc:348] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.457926 24103 raft_consensus.cc:374] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:47.457947 24103 raft_consensus.cc:729] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3b5a8a8d0b43908e833f89161d730a, State: Initialized, Role: FOLLOWER
I20260929 03:30:47.458060 24103 consensus_queue.cc:260] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.458230 24106 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: Bootstrap starting.
I20260929 03:30:47.458330 24103 ts_tablet_manager.cc:1428] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:47.458443 23955 heartbeater.cc:503] Master 127.6.116.190:40717 was elected leader, sending a full tablet report...
I20260929 03:30:47.458818 24106 tablet_bootstrap.cc:654] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:47.458874 24104 tablet_bootstrap.cc:654] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:47.459460 24106 log.cc:824] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:47.459463 24104 log.cc:824] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:47.460098 24106 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: No bootstrap required, opened a new log
I20260929 03:30:47.460119 24104 tablet_bootstrap.cc:492] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: No bootstrap required, opened a new log
I20260929 03:30:47.460162 24104 ts_tablet_manager.cc:1397] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 03:30:47.460177 24106 ts_tablet_manager.cc:1397] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:47.461417 24104 raft_consensus.cc:348] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.461563 24104 raft_consensus.cc:374] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:47.461429 24106 raft_consensus.cc:348] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.461613 24104 raft_consensus.cc:729] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32d4c2a2ed448958257d732aee6b192, State: Initialized, Role: FOLLOWER
I20260929 03:30:47.461635 24106 raft_consensus.cc:374] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:47.461676 24106 raft_consensus.cc:729] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8248f86c05064949993d549fa8f51e4a, State: Initialized, Role: FOLLOWER
I20260929 03:30:47.461757 24104 consensus_queue.cc:260] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.461781 24106 consensus_queue.cc:260] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.462044 24106 ts_tablet_manager.cc:1428] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:47.462067 23821 heartbeater.cc:503] Master 127.6.116.190:40717 was elected leader, sending a full tablet report...
I20260929 03:30:47.462066 24104 ts_tablet_manager.cc:1428] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 03:30:47.462069 24089 heartbeater.cc:503] Master 127.6.116.190:40717 was elected leader, sending a full tablet report...
W20260929 03:30:47.522814 23956 tablet.cc:2369] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:47.564591 24107 raft_consensus.cc:482] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:47.564694 24107 raft_consensus.cc:504] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.565124 24107 leader_election.cc:288] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d32d4c2a2ed448958257d732aee6b192 (127.6.116.131:34115), 8248f86c05064949993d549fa8f51e4a (127.6.116.129:34119)
I20260929 03:30:47.568948 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.565441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:48999 (local address 127.6.116.129:34119)
0929 03:30:47.565494 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.565496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.565642 (+ 146us) server_negotiation.cc:407] Connection header received
0929 03:30:47.565665 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.565666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.565674 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.565861 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:47.566000 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.566686 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568027 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.568609 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568687 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.568695 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.568857 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.568899 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:30:47.568994 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.565277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:34115 (local address 127.6.116.130:48651)
0929 03:30:47.565350 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.565362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.565407 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.565567 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.565569 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.565571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:47.565648 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.565650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.566698 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.566698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.567941 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.567944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.568702 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.568781 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.568783 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.568839 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.568898 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":32}
I20260929 03:30:47.569001 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.565304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34119 (local address 127.6.116.130:48999)
0929 03:30:47.565552 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.565562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.565612 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.565880 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.565880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.565881 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:47.565930 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.565932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.566762 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.566763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.567941 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.567944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.568707 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.568777 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.568781 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.568840 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.568898 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":16,"thread_start_us":109,"threads_started":1}
I20260929 03:30:47.569221 24102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.565372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:48651 (local address 127.6.116.131:34115)
0929 03:30:47.565421 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.565424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.565432 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:47.565462 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.565463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.565474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.565539 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:47.565832 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.566618 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568027 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.568627 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.568686 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.568695 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.569061 (+ 366us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.569113 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:30:47.569481 24043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8df219857340aab280f914bddfb14a" candidate_uuid: "5e3b5a8a8d0b43908e833f89161d730a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32d4c2a2ed448958257d732aee6b192" is_pre_election: true
I20260929 03:30:47.569479 23775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8df219857340aab280f914bddfb14a" candidate_uuid: "5e3b5a8a8d0b43908e833f89161d730a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8248f86c05064949993d549fa8f51e4a" is_pre_election: true
I20260929 03:30:47.569622 23775 raft_consensus.cc:2454] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3b5a8a8d0b43908e833f89161d730a in term 0.
I20260929 03:30:47.569622 24043 raft_consensus.cc:2454] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e3b5a8a8d0b43908e833f89161d730a in term 0.
I20260929 03:30:47.569847 23843 leader_election.cc:302] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3b5a8a8d0b43908e833f89161d730a, 8248f86c05064949993d549fa8f51e4a; no voters:
I20260929 03:30:47.569988 24107 raft_consensus.cc:2781] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:47.570051 24107 raft_consensus.cc:482] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:47.570091 24107 raft_consensus.cc:3037] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:47.570808 24107 raft_consensus.cc:504] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.570945 24107 leader_election.cc:288] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [CANDIDATE]: Term 1 election: Requested vote from peers d32d4c2a2ed448958257d732aee6b192 (127.6.116.131:34115), 8248f86c05064949993d549fa8f51e4a (127.6.116.129:34119)
I20260929 03:30:47.571130 24043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8df219857340aab280f914bddfb14a" candidate_uuid: "5e3b5a8a8d0b43908e833f89161d730a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32d4c2a2ed448958257d732aee6b192"
I20260929 03:30:47.571128 23775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b8df219857340aab280f914bddfb14a" candidate_uuid: "5e3b5a8a8d0b43908e833f89161d730a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8248f86c05064949993d549fa8f51e4a"
I20260929 03:30:47.571208 24043 raft_consensus.cc:3037] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:47.571206 23775 raft_consensus.cc:3037] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:47.572074 24043 raft_consensus.cc:2454] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3b5a8a8d0b43908e833f89161d730a in term 1.
I20260929 03:30:47.572074 23775 raft_consensus.cc:2454] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e3b5a8a8d0b43908e833f89161d730a in term 1.
I20260929 03:30:47.572238 23843 leader_election.cc:302] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e3b5a8a8d0b43908e833f89161d730a, 8248f86c05064949993d549fa8f51e4a; no voters:
I20260929 03:30:47.572359 24107 raft_consensus.cc:2781] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:47.572526 24107 raft_consensus.cc:686] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 1 LEADER]: Becoming Leader. State: Replica: 5e3b5a8a8d0b43908e833f89161d730a, State: Running, Role: LEADER
I20260929 03:30:47.572636 24107 consensus_queue.cc:237] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } }
I20260929 03:30:47.573503 23630 catalog_manager.cc:5495] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e3b5a8a8d0b43908e833f89161d730a (127.6.116.130). New cstate: current_term: 1 leader_uuid: "5e3b5a8a8d0b43908e833f89161d730a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e3b5a8a8d0b43908e833f89161d730a" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 } health_report { overall_health: UNKNOWN } } }
I20260929 03:30:47.593462 24099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.591124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.6.116.130:45323)
0929 03:30:47.591178 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.591180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.591189 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:47.591220 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.591221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.591237 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.591294 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:47.591440 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.591897 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.592666 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.592877 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.592953 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.593141 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:47.593237 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:47.593327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.593350 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.593368 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:30:47.595125 24043 raft_consensus.cc:1261] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 1 FOLLOWER]: Refusing update from remote peer 5e3b5a8a8d0b43908e833f89161d730a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:47.595125 23775 raft_consensus.cc:1261] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 1 FOLLOWER]: Refusing update from remote peer 5e3b5a8a8d0b43908e833f89161d730a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:47.595492 24113 consensus_queue.cc:1035] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32d4c2a2ed448958257d732aee6b192" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 34115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:47.595616 24107 consensus_queue.cc:1035] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8248f86c05064949993d549fa8f51e4a" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:47.597946 24116 mvcc.cc:205] Tried to move back new op lower bound from 7334513244546850816 to 7334513244458139648. Current Snapshot: MvccSnapshot[applied={T|T < 7334513244546850816}]
I20260929 03:30:47.598572 24117 mvcc.cc:205] Tried to move back new op lower bound from 7334513244546850816 to 7334513244458139648. Current Snapshot: MvccSnapshot[applied={T|T < 7334513244546850816}]
I20260929 03:30:47.599337 24118 mvcc.cc:205] Tried to move back new op lower bound from 7334513244546850816 to 7334513244458139648. Current Snapshot: MvccSnapshot[applied={T|T < 7334513244546850816 or (T in {7334513244546850816})}]
I20260929 03:30:47.600476 23629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:47.600569 23629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:47.601266 23629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a66d80c176a84e78822b7a05fb1ec8ab: Sending DeleteTablet for 3 replicas of tablet 8b8df219857340aab280f914bddfb14a
I20260929 03:30:47.601588 23889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8df219857340aab280f914bddfb14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:47 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:55480
I20260929 03:30:47.601605 23755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8df219857340aab280f914bddfb14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:47 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:52212
I20260929 03:30:47.601806 24023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b8df219857340aab280f914bddfb14a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:47 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:60426
I20260929 03:30:47.601866 24125 tablet_replica.cc:330] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a: stopping tablet replica
I20260929 03:30:47.601953 24125 raft_consensus.cc:2229] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:47.602012 24126 tablet_replica.cc:330] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: stopping tablet replica
I20260929 03:30:47.602044 24125 raft_consensus.cc:2258] T 8b8df219857340aab280f914bddfb14a P 8248f86c05064949993d549fa8f51e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:47.602105 24126 raft_consensus.cc:2229] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:47.602162 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23692
I20260929 03:30:47.602200 24126 raft_consensus.cc:2258] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:47.602559 24126 ts_tablet_manager.cc:1905] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:47.602639 24127 tablet_replica.cc:330] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: stopping tablet replica
I20260929 03:30:47.602737 24127 raft_consensus.cc:2229] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:47.602841 24127 raft_consensus.cc:2258] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:47.603310 24127 ts_tablet_manager.cc:1905] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:47.604099 24126 ts_tablet_manager.cc:1918] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:47.604202 24126 log.cc:1196] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/8b8df219857340aab280f914bddfb14a
I20260929 03:30:47.604467 24126 ts_tablet_manager.cc:1939] T 8b8df219857340aab280f914bddfb14a P 5e3b5a8a8d0b43908e833f89161d730a: Deleting consensus metadata
I20260929 03:30:47.604837 23617 catalog_manager.cc:4841] TS 5e3b5a8a8d0b43908e833f89161d730a (127.6.116.130:45323): tablet 8b8df219857340aab280f914bddfb14a (table test-table [id=e8a0200d17d44e86ba4fc4387f4be4c0]) successfully deleted
I20260929 03:30:47.605189 24127 ts_tablet_manager.cc:1918] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:47.605278 24127 log.cc:1196] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/8b8df219857340aab280f914bddfb14a
I20260929 03:30:47.605531 24127 ts_tablet_manager.cc:1939] T 8b8df219857340aab280f914bddfb14a P d32d4c2a2ed448958257d732aee6b192: Deleting consensus metadata
I20260929 03:30:47.605967 23617 catalog_manager.cc:4841] TS d32d4c2a2ed448958257d732aee6b192 (127.6.116.131:34115): tablet 8b8df219857340aab280f914bddfb14a (table test-table [id=e8a0200d17d44e86ba4fc4387f4be4c0]) successfully deleted
W20260929 03:30:47.610121 23615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:34119 (error 108)
I20260929 03:30:47.610616 24101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.610475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34119 (local address 127.0.0.1:52222)
0929 03:30:47.610541 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.610584 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:30:47.610735 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23826
W20260929 03:30:47.610731 23615 catalog_manager.cc:4584] TS 8248f86c05064949993d549fa8f51e4a (127.6.116.129:34119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b8df219857340aab280f914bddfb14a: Network error: Client connection negotiation failed: client connection to 127.6.116.129:34119: connect: Connection refused (error 111)
I20260929 03:30:47.615224 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23960
I20260929 03:30:47.619453 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 23598
2026-09-29T03:30:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1901 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T03:30:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:47Z Disabled control of system clock
I20260929 03:30:47.646121 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39479
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:39479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:47.723688 24132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.723886 24132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.723907 24132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.724968 24132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:47.725003 24132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.725016 24132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.725029 24132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:47.725041 24132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:47.726296 24132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:39479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.726473 24132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.726687 24132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.731518 24137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.731525 24140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.731525 24138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.732146 24132 server_base.cc:1034] running on GCE node
I20260929 03:30:47.732350 24132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.732580 24132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.733734 24132 hybrid_clock.cc:648] HybridClock initialized: now 1790652647733708 us; error 30 us; skew 500 ppm
I20260929 03:30:47.734918 24132 webserver.cc:460] Webserver started at http://127.6.116.190:39217/ using document root <none> and password file <none>
I20260929 03:30:47.735189 24132 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.735237 24132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.735322 24132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.736261 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "722d0d0636884a2dbd12f11ef6136fbe"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "d74ffd403f9a39fd261fafdefb9e2260"
server_key_iv: "800c76f0dc2c00d424f18dcd326c562a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.736541 24132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "722d0d0636884a2dbd12f11ef6136fbe"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "d74ffd403f9a39fd261fafdefb9e2260"
server_key_iv: "800c76f0dc2c00d424f18dcd326c562a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.737709 24132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:30:47.738344 24146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.738513 24132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:47.738603 24132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "722d0d0636884a2dbd12f11ef6136fbe"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "d74ffd403f9a39fd261fafdefb9e2260"
server_key_iv: "800c76f0dc2c00d424f18dcd326c562a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.738685 24132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:47.756033 24132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:47.756310 24132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:47.756464 24132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:47.760275 24132 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:39479
I20260929 03:30:47.760305 24198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:39479 every 8 connection(s)
I20260929 03:30:47.760613 24132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:47.761484 24199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:47.763633 24199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: Bootstrap starting.
I20260929 03:30:47.764123 24199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:47.764344 24199 log.cc:824] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:47.764899 24199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: No bootstrap required, opened a new log
I20260929 03:30:47.766350 24199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } }
I20260929 03:30:47.766472 24199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:47.766494 24199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722d0d0636884a2dbd12f11ef6136fbe, State: Initialized, Role: FOLLOWER
I20260929 03:30:47.766587 24199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } }
I20260929 03:30:47.766646 24199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:47.766667 24199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:47.766723 24199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:47.767268 24199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } }
I20260929 03:30:47.767373 24199 leader_election.cc:302] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722d0d0636884a2dbd12f11ef6136fbe; no voters:
I20260929 03:30:47.767536 24199 leader_election.cc:288] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:47.767578 24202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:47.767745 24202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [term 1 LEADER]: Becoming Leader. State: Replica: 722d0d0636884a2dbd12f11ef6136fbe, State: Running, Role: LEADER
I20260929 03:30:47.767786 24199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:47.767866 24202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } }
I20260929 03:30:47.768388 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722d0d0636884a2dbd12f11ef6136fbe. Latest consensus state: current_term: 1 leader_uuid: "722d0d0636884a2dbd12f11ef6136fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } } }
I20260929 03:30:47.768482 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:47.768699 24203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722d0d0636884a2dbd12f11ef6136fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722d0d0636884a2dbd12f11ef6136fbe" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 39479 } } }
I20260929 03:30:47.768779 24203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:47.768784 24211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:47.769245 24211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:47.770133 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24132
I20260929 03:30:47.770262 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:47.770489 6610 external_mini_cluster.cc:1428] Setting key fd65d76a15b013d70c3585f4d1b4084a
I20260929 03:30:47.771034 24211 catalog_manager.cc:1347] Generated new cluster ID: d7d65e1c9c0142c09149901cec143123
I20260929 03:30:47.771109 24211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:47.775596 24211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:47.776274 24211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:47.779640 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.770902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.6.116.190:39479)
0929 03:30:47.775414 (+ 4512us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.775420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.775443 (+ 23us) server_negotiation.cc:407] Connection header received
0929 03:30:47.775569 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.775577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.775654 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.775790 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.775992 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.776951 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.777722 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.778106 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.778238 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.778923 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.778926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.778926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:47.779169 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.779202 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:47.779212 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.779297 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.779368 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.779451 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4461,"thread_start_us":68,"threads_started":1}
I20260929 03:30:47.783279 24211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: Generated new TSK 0
I20260929 03:30:47.783427 24211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:47.784345 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--builtin_ntp_servers=127.6.116.148:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:47.864604 24223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.864820 24223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.864876 24223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.866070 24223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.866134 24223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.866174 24223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:47.867832 24223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.868129 24223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.868422 24223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.873471 24228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.873494 24229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.873498 24231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.873719 24223 server_base.cc:1034] running on GCE node
I20260929 03:30:47.873904 24223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.874169 24223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.875334 24223 hybrid_clock.cc:648] HybridClock initialized: now 1790652647875320 us; error 32 us; skew 500 ppm
I20260929 03:30:47.876580 24223 webserver.cc:460] Webserver started at http://127.6.116.129:46125/ using document root <none> and password file <none>
I20260929 03:30:47.876827 24223 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.876899 24223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.877040 24223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.878074 24223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "353304a18e4d4d3e905f0b4829241b84"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "58f4a3c711a1cc7b02506e61383107f6"
server_key_iv: "74f9bbec73a48cbf3c2d23ecad2f3822"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.878428 24223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "353304a18e4d4d3e905f0b4829241b84"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "58f4a3c711a1cc7b02506e61383107f6"
server_key_iv: "74f9bbec73a48cbf3c2d23ecad2f3822"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.879856 24223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:47.880662 24237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.880838 24223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:30:47.880935 24223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "353304a18e4d4d3e905f0b4829241b84"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "58f4a3c711a1cc7b02506e61383107f6"
server_key_iv: "74f9bbec73a48cbf3c2d23ecad2f3822"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.881018 24223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:47.890913 24223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:47.891215 24223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:47.891368 24223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:47.891583 24223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:47.891889 24223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:47.891945 24223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.891981 24223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:47.892007 24223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.898142 24223 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:39179
I20260929 03:30:47.898186 24350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:39179 every 8 connection(s)
I20260929 03:30:47.898511 24223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:47.899618 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24223
I20260929 03:30:47.899720 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:47.899912 6610 external_mini_cluster.cc:1428] Setting key 72de89ed3b8be651287a444b121b2ddc
I20260929 03:30:47.900908 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--builtin_ntp_servers=127.6.116.148:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:30:47.904515 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.899210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:57901 (local address 127.6.116.190:39479)
0929 03:30:47.899250 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:47.899252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:47.899503 (+ 251us) server_negotiation.cc:407] Connection header received
0929 03:30:47.900205 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:47.900206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:47.900219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:47.900269 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:47.901116 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.901893 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.903031 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.903354 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.903426 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.904256 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:47.904257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:47.904257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:47.904262 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:47.904277 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:47.904281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:47.904336 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:47.904406 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:30:47.904440 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:30:47.904793 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:47.899203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39479 (local address 127.6.116.129:57901)
0929 03:30:47.899371 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:47.899412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:47.900087 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:47.900346 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:47.900350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:47.900361 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:47.901009 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.901016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.902006 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.902007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.902917 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:47.902920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:47.903435 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:47.903436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:47.903552 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:47.904103 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:47.904103 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:47.904130 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:47.904149 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:47.904339 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:47.904344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:47.904347 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:47.904411 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:30:47.904503 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260929 03:30:47.905328 24351 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39479
I20260929 03:30:47.905403 24351 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:47.905576 24351 heartbeater.cc:511] Master 127.6.116.190:39479 requested a full tablet report, sending...
I20260929 03:30:47.906073 24163 ts_manager.cc:194] Registered new tserver with Master: 353304a18e4d4d3e905f0b4829241b84 (127.6.116.129:39179)
I20260929 03:30:47.906659 24163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.129:57901
W20260929 03:30:47.979221 24354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:47.979416 24354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:47.979453 24354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:47.980562 24354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:47.980621 24354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:47.980662 24354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:47.981968 24354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:47.982213 24354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:47.982481 24354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:47.987171 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.987180 24360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:47.987182 24359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:47.987457 24354 server_base.cc:1034] running on GCE node
I20260929 03:30:47.987664 24354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:47.987879 24354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:47.989013 24354 hybrid_clock.cc:648] HybridClock initialized: now 1790652647989001 us; error 31 us; skew 500 ppm
I20260929 03:30:47.990094 24354 webserver.cc:460] Webserver started at http://127.6.116.130:45429/ using document root <none> and password file <none>
I20260929 03:30:47.990304 24354 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:47.990347 24354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:47.990499 24354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:47.991680 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "3e98082e7fd64b02b66c6cd63d2f5269"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "f4297cb945947164921414a7cee1f3d7"
server_key_iv: "4dd02adacf6bf4ffbb32011be8ee301a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.992079 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "3e98082e7fd64b02b66c6cd63d2f5269"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "f4297cb945947164921414a7cee1f3d7"
server_key_iv: "4dd02adacf6bf4ffbb32011be8ee301a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.993399 24354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:47.994094 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:47.994266 24354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:47.994361 24354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "3e98082e7fd64b02b66c6cd63d2f5269"
format_stamp: "Formatted at 2026-09-29 03:30:47 on dist-test-slave-z2zg"
server_key: "f4297cb945947164921414a7cee1f3d7"
server_key_iv: "4dd02adacf6bf4ffbb32011be8ee301a"
server_key_version: "encryptionkey@0"
I20260929 03:30:47.994441 24354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:48.009717 24354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:48.009965 24354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:48.010123 24354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:48.010317 24354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:48.010607 24354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:48.010661 24354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:48.010701 24354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:48.010728 24354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:48.016068 24354 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:34263
I20260929 03:30:48.016175 24481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:34263 every 8 connection(s)
I20260929 03:30:48.016426 24354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:48.020978 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.017079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:46161 (local address 127.6.116.190:39479)
0929 03:30:48.017163 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.017165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.017335 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:30:48.017730 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.017731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.017744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.017795 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:48.018361 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.018836 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.019734 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.019921 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.019971 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.020783 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:48.020783 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:48.020784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:48.020789 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:48.020798 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:48.020802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:48.020839 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.020904 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.020924 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 03:30:48.021145 24484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.017051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39479 (local address 127.6.116.130:46161)
0929 03:30:48.017231 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.017258 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.017653 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.017841 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.017844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.017851 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:48.018295 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.018298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.018914 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.018915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.019664 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.019666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.019992 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.019993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.020088 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.020660 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:48.020660 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:48.020694 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:48.020715 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:48.020841 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:48.020843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:48.020845 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.020891 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.020957 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260929 03:30:48.021610 24482 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39479
I20260929 03:30:48.021688 24482 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:48.021808 24482 heartbeater.cc:511] Master 127.6.116.190:39479 requested a full tablet report, sending...
I20260929 03:30:48.022168 24163 ts_manager.cc:194] Registered new tserver with Master: 3e98082e7fd64b02b66c6cd63d2f5269 (127.6.116.130:34263)
I20260929 03:30:48.022516 24163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.130:46161
I20260929 03:30:48.025782 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24354
I20260929 03:30:48.025885 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:48.026075 6610 external_mini_cluster.cc:1428] Setting key de0356936fbe5b4eb83e3e8de4cbd9fd
I20260929 03:30:48.026875 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--builtin_ntp_servers=127.6.116.148:46249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:48.104238 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:48.104413 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:48.104432 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:48.105530 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:48.105567 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:48.105592 24485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:48.106927 24485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:39479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:48.107167 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:48.107431 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:48.112255 24491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:48.112269 24490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:48.112269 24485 server_base.cc:1034] running on GCE node
W20260929 03:30:48.112390 24493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:48.112622 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:48.112866 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:48.114019 24485 hybrid_clock.cc:648] HybridClock initialized: now 1790652648114002 us; error 29 us; skew 500 ppm
I20260929 03:30:48.115207 24485 webserver.cc:460] Webserver started at http://127.6.116.131:45859/ using document root <none> and password file <none>
I20260929 03:30:48.115434 24485 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:48.115499 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:48.115634 24485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:48.116607 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "2c6313252a1d49fcbb2791d0f57e7703"
format_stamp: "Formatted at 2026-09-29 03:30:48 on dist-test-slave-z2zg"
server_key: "e2a36a00506290a683a934c44af51959"
server_key_iv: "e80e66ef6c9443a32717400320881e0e"
server_key_version: "encryptionkey@0"
I20260929 03:30:48.116962 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "2c6313252a1d49fcbb2791d0f57e7703"
format_stamp: "Formatted at 2026-09-29 03:30:48 on dist-test-slave-z2zg"
server_key: "e2a36a00506290a683a934c44af51959"
server_key_iv: "e80e66ef6c9443a32717400320881e0e"
server_key_version: "encryptionkey@0"
I20260929 03:30:48.118172 24485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:48.118839 24499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:48.118994 24485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:48.119114 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "2c6313252a1d49fcbb2791d0f57e7703"
format_stamp: "Formatted at 2026-09-29 03:30:48 on dist-test-slave-z2zg"
server_key: "e2a36a00506290a683a934c44af51959"
server_key_iv: "e80e66ef6c9443a32717400320881e0e"
server_key_version: "encryptionkey@0"
I20260929 03:30:48.119200 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:48.125546 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:48.125825 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:48.125967 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:48.126178 24485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:48.126466 24485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:48.126518 24485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:48.126557 24485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:48.126585 24485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:48.132035 24485 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:41761
I20260929 03:30:48.132071 24612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:41761 every 8 connection(s)
I20260929 03:30:48.132354 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:30:48.137233 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.133100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:42611 (local address 127.6.116.190:39479)
0929 03:30:48.133165 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.133167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.133347 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:30:48.133808 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.133810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.133826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.133871 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:48.134466 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.134938 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.135962 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.136159 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.136233 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.136953 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:48.136955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:48.136955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:48.136959 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:48.136971 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:48.136976 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:48.137041 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.137133 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.137159 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:30:48.137445 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.133084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:39479 (local address 127.6.116.131:42611)
0929 03:30:48.133248 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.133274 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.133701 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.133940 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.133943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.133958 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:48.134379 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.134383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.135030 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.135031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.135864 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.135866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.136238 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.136239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.136359 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.136812 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:48.136812 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:48.136836 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:48.136855 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:48.137042 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:48.137048 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:48.137050 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.137133 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.137219 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260929 03:30:48.137969 24613 heartbeater.cc:347] Connected to a master server at 127.6.116.190:39479
I20260929 03:30:48.138038 24613 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:48.138208 24613 heartbeater.cc:511] Master 127.6.116.190:39479 requested a full tablet report, sending...
I20260929 03:30:48.138545 24163 ts_manager.cc:194] Registered new tserver with Master: 2c6313252a1d49fcbb2791d0f57e7703 (127.6.116.131:41761)
I20260929 03:30:48.138857 24163 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.131:42611
I20260929 03:30:48.141206 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24485
I20260929 03:30:48.141284 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:48.141412 6610 external_mini_cluster.cc:1428] Setting key c889402a7a48ba8ca9831eee60df3373
I20260929 03:30:48.142313 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:48.146005 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.143600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.6.116.190:39479)
0929 03:30:48.143665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.143668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.143752 (+ 84us) server_negotiation.cc:407] Connection header received
0929 03:30:48.143787 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.143788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.143795 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.143836 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:48.143956 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.144372 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.145038 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.145203 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.145249 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.145756 (+ 507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:48.145757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:48.145757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:48.145761 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:48.145771 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:48.145774 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:48.145815 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.145873 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.145890 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:30:48.147392 24163 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:48.148816 24163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:48.154554 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.150840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39179 (local address 127.0.0.1:44748)
0929 03:30:48.151087 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.151095 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.151136 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.151379 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.151381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.151384 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:48.151441 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.151443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152418 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.152419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.153520 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.153523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154290 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.154291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.154365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.154377 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.154444 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.154488 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":85,"threads_started":1}
I20260929 03:30:48.154568 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.150943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.6.116.129:39179)
0929 03:30:48.151181 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.151185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.151197 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:30:48.151236 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.151239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.151255 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.151376 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:48.151509 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152322 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.153612 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154193 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.154285 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.154366 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.154461 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.154497 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260929 03:30:48.154878 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.150812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:34263 (local address 127.0.0.1:48348)
0929 03:30:48.150994 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.151031 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.151118 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.151425 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.151425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.151427 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:48.151455 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.151457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152828 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.152829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.153822 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.153824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154671 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.154673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.154742 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.154744 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.154770 (+ 26us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.154805 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":54,"threads_started":1}
I20260929 03:30:48.155005 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.151019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:41761 (local address 127.0.0.1:41192)
0929 03:30:48.151314 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.151325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.151376 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.151708 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.151709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.151711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:48.151740 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.151741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152858 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.152859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.153615 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.153617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154518 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.154519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.154581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.154585 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.154880 (+ 295us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.154921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":68,"threads_started":1}
I20260929 03:30:48.155260 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.151172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.6.116.131:41761)
0929 03:30:48.151301 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.151304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.151526 (+ 222us) server_negotiation.cc:407] Connection header received
0929 03:30:48.151562 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.151564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.151577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.151659 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:48.151812 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152803 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.153698 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154346 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.154412 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.154484 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.155149 (+ 665us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.155187 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260929 03:30:48.155397 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.150908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.6.116.130:34263)
0929 03:30:48.151267 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.151271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.151282 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:30:48.151320 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.151322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.151336 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.151552 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:48.151615 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.152736 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.153902 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.154565 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.155089 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.155246 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.155271 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.155313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":79,"threads_started":1}
I20260929 03:30:48.155295 24285 tablet_service.cc:1467] Processing CreateTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 (DEFAULT_TABLE table=test-table [id=fbedccc03c5e4a149fd74642c9af9188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:48.155639 24285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2af35c4d89345a8b6ed4ba545c68ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:48.155821 24547 tablet_service.cc:1467] Processing CreateTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 (DEFAULT_TABLE table=test-table [id=fbedccc03c5e4a149fd74642c9af9188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:48.155972 24416 tablet_service.cc:1467] Processing CreateTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 (DEFAULT_TABLE table=test-table [id=fbedccc03c5e4a149fd74642c9af9188]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:48.156143 24547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2af35c4d89345a8b6ed4ba545c68ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:48.156270 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2af35c4d89345a8b6ed4ba545c68ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:48.158324 24627 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Bootstrap starting.
I20260929 03:30:48.159128 24627 tablet_bootstrap.cc:654] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:48.159185 24628 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: Bootstrap starting.
I20260929 03:30:48.159441 24627 log.cc:824] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:48.159457 24629 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Bootstrap starting.
I20260929 03:30:48.159811 24628 tablet_bootstrap.cc:654] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:48.160059 24629 tablet_bootstrap.cc:654] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:48.160141 24628 log.cc:824] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:48.160316 24627 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: No bootstrap required, opened a new log
I20260929 03:30:48.160383 24627 ts_tablet_manager.cc:1397] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 03:30:48.160382 24629 log.cc:824] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:48.160907 24628 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: No bootstrap required, opened a new log
I20260929 03:30:48.160998 24629 tablet_bootstrap.cc:492] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: No bootstrap required, opened a new log
I20260929 03:30:48.161049 24629 ts_tablet_manager.cc:1397] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:48.161118 24628 ts_tablet_manager.cc:1397] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:48.162843 24629 raft_consensus.cc:348] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.162957 24629 raft_consensus.cc:374] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:48.162978 24629 raft_consensus.cc:729] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e98082e7fd64b02b66c6cd63d2f5269, State: Initialized, Role: FOLLOWER
I20260929 03:30:48.162974 24627 raft_consensus.cc:348] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.163105 24629 consensus_queue.cc:260] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.163168 24627 raft_consensus.cc:374] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:48.163210 24627 raft_consensus.cc:729] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6313252a1d49fcbb2791d0f57e7703, State: Initialized, Role: FOLLOWER
I20260929 03:30:48.163352 24629 ts_tablet_manager.cc:1428] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 03:30:48.163343 24627 consensus_queue.cc:260] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.163491 24482 heartbeater.cc:503] Master 127.6.116.190:39479 was elected leader, sending a full tablet report...
I20260929 03:30:48.163566 24627 ts_tablet_manager.cc:1428] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 03:30:48.163672 24613 heartbeater.cc:503] Master 127.6.116.190:39479 was elected leader, sending a full tablet report...
I20260929 03:30:48.163938 24628 raft_consensus.cc:348] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.164081 24628 raft_consensus.cc:374] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:48.164113 24628 raft_consensus.cc:729] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353304a18e4d4d3e905f0b4829241b84, State: Initialized, Role: FOLLOWER
I20260929 03:30:48.164250 24628 consensus_queue.cc:260] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.164481 24628 ts_tablet_manager.cc:1428] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:48.164585 24351 heartbeater.cc:503] Master 127.6.116.190:39479 was elected leader, sending a full tablet report...
W20260929 03:30:48.266934 24483 tablet.cc:2369] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:48.343990 24635 raft_consensus.cc:482] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:48.344098 24635 raft_consensus.cc:504] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.344413 24635 leader_election.cc:288] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e98082e7fd64b02b66c6cd63d2f5269 (127.6.116.130:34263), 2c6313252a1d49fcbb2791d0f57e7703 (127.6.116.131:41761)
I20260929 03:30:48.348095 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.344668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:53165 (local address 127.6.116.130:34263)
0929 03:30:48.344777 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.344780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.344789 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:48.344818 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.344819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.344829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.344888 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:48.345062 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.345828 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347237 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.347761 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.347839 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.347993 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.348028 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 03:30:48.348095 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.344769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34967 (local address 127.6.116.131:41761)
0929 03:30:48.344812 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.344814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.344985 (+ 171us) server_negotiation.cc:407] Connection header received
0929 03:30:48.345010 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.345011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.345018 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.345081 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:48.345220 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.345893 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347236 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.347756 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347831 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.347839 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.347993 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.348028 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:30:48.348124 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.344592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:34263 (local address 127.6.116.129:53165)
0929 03:30:48.344659 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.344672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.344726 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.344916 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.344918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.344919 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:48.344995 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.344997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.345906 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.345906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.347152 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.347156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.347840 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.347916 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.347920 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.347976 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.348020 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":14}
I20260929 03:30:48.348310 24636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.344711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:41761 (local address 127.6.116.129:34967)
0929 03:30:48.344868 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.344894 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:48.344946 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:48.345094 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:48.345095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:48.345096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:48.345159 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.345160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.345958 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.345959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.347156 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:48.347157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.347841 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.347842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:48.347918 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.347920 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:48.347976 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:30:48.348018 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260929 03:30:48.348783 24567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2af35c4d89345a8b6ed4ba545c68ad5" candidate_uuid: "353304a18e4d4d3e905f0b4829241b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6313252a1d49fcbb2791d0f57e7703" is_pre_election: true
I20260929 03:30:48.348778 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2af35c4d89345a8b6ed4ba545c68ad5" candidate_uuid: "353304a18e4d4d3e905f0b4829241b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" is_pre_election: true
I20260929 03:30:48.348922 24567 raft_consensus.cc:2454] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353304a18e4d4d3e905f0b4829241b84 in term 0.
I20260929 03:30:48.348922 24436 raft_consensus.cc:2454] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 353304a18e4d4d3e905f0b4829241b84 in term 0.
I20260929 03:30:48.349136 24238 leader_election.cc:302] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353304a18e4d4d3e905f0b4829241b84, 3e98082e7fd64b02b66c6cd63d2f5269; no voters:
I20260929 03:30:48.349251 24635 raft_consensus.cc:2781] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:48.349304 24635 raft_consensus.cc:482] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:48.349328 24635 raft_consensus.cc:3037] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:48.349972 24635 raft_consensus.cc:504] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.350092 24635 leader_election.cc:288] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [CANDIDATE]: Term 1 election: Requested vote from peers 3e98082e7fd64b02b66c6cd63d2f5269 (127.6.116.130:34263), 2c6313252a1d49fcbb2791d0f57e7703 (127.6.116.131:41761)
I20260929 03:30:48.350279 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2af35c4d89345a8b6ed4ba545c68ad5" candidate_uuid: "353304a18e4d4d3e905f0b4829241b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e98082e7fd64b02b66c6cd63d2f5269"
I20260929 03:30:48.350294 24567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2af35c4d89345a8b6ed4ba545c68ad5" candidate_uuid: "353304a18e4d4d3e905f0b4829241b84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6313252a1d49fcbb2791d0f57e7703"
I20260929 03:30:48.350359 24567 raft_consensus.cc:3037] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:48.350360 24436 raft_consensus.cc:3037] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:48.351269 24567 raft_consensus.cc:2454] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353304a18e4d4d3e905f0b4829241b84 in term 1.
I20260929 03:30:48.351269 24436 raft_consensus.cc:2454] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 353304a18e4d4d3e905f0b4829241b84 in term 1.
I20260929 03:30:48.351476 24238 leader_election.cc:302] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6313252a1d49fcbb2791d0f57e7703, 353304a18e4d4d3e905f0b4829241b84; no voters:
I20260929 03:30:48.351572 24635 raft_consensus.cc:2781] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:48.351707 24635 raft_consensus.cc:686] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [term 1 LEADER]: Becoming Leader. State: Replica: 353304a18e4d4d3e905f0b4829241b84, State: Running, Role: LEADER
I20260929 03:30:48.351804 24635 consensus_queue.cc:237] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } }
I20260929 03:30:48.352665 24162 catalog_manager.cc:5495] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 353304a18e4d4d3e905f0b4829241b84 (127.6.116.129). New cstate: current_term: 1 leader_uuid: "353304a18e4d4d3e905f0b4829241b84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "353304a18e4d4d3e905f0b4829241b84" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 39179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 } health_report { overall_health: UNKNOWN } } }
I20260929 03:30:48.378572 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.376412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.6.116.129:39179)
0929 03:30:48.376474 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:48.376476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:48.376485 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:48.376518 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:48.376519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:48.376542 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:48.376635 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:48.376756 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.377178 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.377847 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:48.378019 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:48.378094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:48.378263 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:48.378347 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:48.378424 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:48.378462 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:30:48.378479 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:30:48.380348 24436 raft_consensus.cc:1261] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 1 FOLLOWER]: Refusing update from remote peer 353304a18e4d4d3e905f0b4829241b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:48.380348 24567 raft_consensus.cc:1261] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 1 FOLLOWER]: Refusing update from remote peer 353304a18e4d4d3e905f0b4829241b84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:48.380709 24635 consensus_queue.cc:1035] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e98082e7fd64b02b66c6cd63d2f5269" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 34263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:48.380841 24637 consensus_queue.cc:1035] T d2af35c4d89345a8b6ed4ba545c68ad5 P 353304a18e4d4d3e905f0b4829241b84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6313252a1d49fcbb2791d0f57e7703" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 41761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 03:30:48.382820 24614 tablet.cc:2369] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:48.383018 24640 mvcc.cc:205] Tried to move back new op lower bound from 7334513247763017728 to 7334513247649628160. Current Snapshot: MvccSnapshot[applied={T|T < 7334513247763017728}]
I20260929 03:30:48.383301 24642 mvcc.cc:205] Tried to move back new op lower bound from 7334513247763017728 to 7334513247649628160. Current Snapshot: MvccSnapshot[applied={T|T < 7334513247763017728}]
I20260929 03:30:48.384064 24641 mvcc.cc:205] Tried to move back new op lower bound from 7334513247763017728 to 7334513247649628160. Current Snapshot: MvccSnapshot[applied={T|T < 7334513247763017728}]
I20260929 03:30:48.385808 24163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:48.385903 24163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:48.386582 24163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 722d0d0636884a2dbd12f11ef6136fbe: Sending DeleteTablet for 3 replicas of tablet d2af35c4d89345a8b6ed4ba545c68ad5
I20260929 03:30:48.386883 24285 tablet_service.cc:1514] Processing DeleteTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:48 UTC) from {username='slave'} at 127.0.0.1:44748
I20260929 03:30:48.386942 24416 tablet_service.cc:1514] Processing DeleteTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:48 UTC) from {username='slave'} at 127.0.0.1:48348
I20260929 03:30:48.386973 24547 tablet_service.cc:1514] Processing DeleteTablet for tablet d2af35c4d89345a8b6ed4ba545c68ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:48 UTC) from {username='slave'} at 127.0.0.1:41192
I20260929 03:30:48.387157 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24223
I20260929 03:30:48.391158 24652 tablet_replica.cc:330] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: stopping tablet replica
I20260929 03:30:48.391268 24652 raft_consensus.cc:2229] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:48.391378 24652 raft_consensus.cc:2258] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:48.391537 24651 tablet_replica.cc:330] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: stopping tablet replica
I20260929 03:30:48.391638 24651 raft_consensus.cc:2229] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:48.391741 24651 raft_consensus.cc:2258] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:48.391813 24652 ts_tablet_manager.cc:1905] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:48.392122 24651 ts_tablet_manager.cc:1905] T d2af35c4d89345a8b6ed4ba545c68ad5 P 3e98082e7fd64b02b66c6cd63d2f5269: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 03:30:48.392445 24148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:39179 (error 108)
I20260929 03:30:48.392570 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24354
I20260929 03:30:48.393469 24652 ts_tablet_manager.cc:1918] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:48.393499 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.393345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:39179 (local address 127.0.0.1:44768)
0929 03:30:48.393421 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.393461 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 03:30:48.393549 24652 log.cc:1196] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/d2af35c4d89345a8b6ed4ba545c68ad5
W20260929 03:30:48.393601 24148 catalog_manager.cc:4584] TS 353304a18e4d4d3e905f0b4829241b84 (127.6.116.129:39179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2af35c4d89345a8b6ed4ba545c68ad5: Network error: Client connection negotiation failed: client connection to 127.6.116.129:39179: connect: Connection refused (error 111)
I20260929 03:30:48.393797 24652 ts_tablet_manager.cc:1939] T d2af35c4d89345a8b6ed4ba545c68ad5 P 2c6313252a1d49fcbb2791d0f57e7703: Deleting consensus metadata
I20260929 03:30:48.394429 24147 catalog_manager.cc:4841] TS 2c6313252a1d49fcbb2791d0f57e7703 (127.6.116.131:41761): tablet d2af35c4d89345a8b6ed4ba545c68ad5 (table test-table [id=fbedccc03c5e4a149fd74642c9af9188]) successfully deleted
I20260929 03:30:48.398463 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:48.398357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:34263 (local address 127.0.0.1:48354)
0929 03:30:48.398406 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:48.398429 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:30:48.398449 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24485
I20260929 03:30:48.402966 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24132
2026-09-29T03:30:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:48 dist-test-slave-z2zg krb5kdc[24656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:48 dist-test-slave-z2zg krb5kdc[24656](info): set up 2 sockets
Sep 29 03:30:48 dist-test-slave-z2zg krb5kdc[24656](info): commencing operation
krb5kdc: starting...
W20260929 03:30:49.408972 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:30:49.517534 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34619
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:34619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:49.593739 24669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:49.593950 24669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:49.593986 24669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:49.595144 24669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:49.595199 24669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:49.595222 24669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:49.595250 24669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:49.595276 24669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:49.596743 24669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:34619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:49.596997 24669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:49.597249 24669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:49.601831 24675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:49.601823 24677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:49.601825 24674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:49.602106 24669 server_base.cc:1034] running on GCE node
I20260929 03:30:49.602334 24669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:49.602607 24669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:49.603767 24669 hybrid_clock.cc:648] HybridClock initialized: now 1790652649603747 us; error 32 us; skew 500 ppm
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:49.606004 24669 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:30:49.606382 24669 webserver.cc:460] Webserver started at http://127.6.116.190:33663/ using document root <none> and password file <none>
I20260929 03:30:49.606580 24669 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:49.606648 24669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:49.606782 24669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:49.607798 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "4b075ebb347b4088b93b566c7c24dd34"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "06badc8fe045ab6a9af5f4739798ebb8"
server_key_iv: "6c487f8d4229d1942d6533ff3d72bd6e"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.608157 24669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "4b075ebb347b4088b93b566c7c24dd34"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "06badc8fe045ab6a9af5f4739798ebb8"
server_key_iv: "6c487f8d4229d1942d6533ff3d72bd6e"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.609372 24669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:49.610039 24684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.610196 24669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:49.610306 24669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "4b075ebb347b4088b93b566c7c24dd34"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "06badc8fe045ab6a9af5f4739798ebb8"
server_key_iv: "6c487f8d4229d1942d6533ff3d72bd6e"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.610379 24669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:49.630263 24669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:49.632004 24669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:49.632151 24669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:49.635728 24669 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:34619
I20260929 03:30:49.635775 24736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:34619 every 8 connection(s)
I20260929 03:30:49.636109 24669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:49.636852 24737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:49.639034 24737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: Bootstrap starting.
I20260929 03:30:49.639793 24737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:49.640117 24737 log.cc:824] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:49.640820 24737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: No bootstrap required, opened a new log
I20260929 03:30:49.641472 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24669
I20260929 03:30:49.641577 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:49.641760 6610 external_mini_cluster.cc:1428] Setting key 2c90f6a5ca6f8140b0dfde59bdb2c192
I20260929 03:30:49.642714 24737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } }
I20260929 03:30:49.642867 24737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:49.642906 24737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b075ebb347b4088b93b566c7c24dd34, State: Initialized, Role: FOLLOWER
I20260929 03:30:49.643061 24737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } }
I20260929 03:30:49.643164 24737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:49.643206 24737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:49.643261 24737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:49.644189 24737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } }
I20260929 03:30:49.644304 24737 leader_election.cc:302] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b075ebb347b4088b93b566c7c24dd34; no voters:
I20260929 03:30:49.644486 24737 leader_election.cc:288] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:49.644508 24742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:49.644618 24742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [term 1 LEADER]: Becoming Leader. State: Replica: 4b075ebb347b4088b93b566c7c24dd34, State: Running, Role: LEADER
I20260929 03:30:49.644724 24737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:49.644696 24742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } }
I20260929 03:30:49.645026 24742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b075ebb347b4088b93b566c7c24dd34. Latest consensus state: current_term: 1 leader_uuid: "4b075ebb347b4088b93b566c7c24dd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } } }
I20260929 03:30:49.645099 24742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:49.645036 24743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b075ebb347b4088b93b566c7c24dd34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b075ebb347b4088b93b566c7c24dd34" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 34619 } } }
I20260929 03:30:49.645257 24743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:49.645314 24747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:49.645823 24747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:49.647653 24747 catalog_manager.cc:1347] Generated new cluster ID: e9c17241c0064455bc6d665158f79123
I20260929 03:30:49.647709 24747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:49.653191 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:49.642212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.6.116.190:34619)
0929 03:30:49.642379 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:49.642386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:49.642408 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:30:49.642697 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:49.642702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:49.642742 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:49.642843 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:49.643695 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.644705 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.645743 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.646414 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.646809 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:49.649306 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:49.649310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:49.649310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:49.649320 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:49.651205 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.651610 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.651612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.651612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.651747 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.651961 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.651961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.651962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.652240 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:49.652804 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:49.652872 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 03:30:49.653001 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260929 03:30:49.670360 24747 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:49.670876 24747 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:49.673626 24747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: Generated new TSK 0
I20260929 03:30:49.673791 24747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:30:49.699975 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:49.789986 24763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:49.790187 24763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:49.790223 24763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:49.791385 24763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:49.791443 24763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:49.791489 24763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:49.792881 24763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:49.793105 24763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:49.793299 24763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:49.798079 24769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:49.798091 24768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:49.798105 24763 server_base.cc:1034] running on GCE node
W20260929 03:30:49.798079 24771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:49.798482 24763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:49.798712 24763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:49.799844 24763 hybrid_clock.cc:648] HybridClock initialized: now 1790652649799830 us; error 31 us; skew 500 ppm
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:49.802184 24763 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:30:49.802724 24763 webserver.cc:460] Webserver started at http://127.6.116.129:46133/ using document root <none> and password file <none>
I20260929 03:30:49.802944 24763 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:49.803006 24763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:49.803165 24763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:49.804090 24763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "52e1f54f5c4a456f9904452e3ccf4b15"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "e7ae3cde91f0079c76939217be019235"
server_key_iv: "721be33d97fbf2b3d49f494b2ab6b8bd"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.804519 24763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "52e1f54f5c4a456f9904452e3ccf4b15"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "e7ae3cde91f0079c76939217be019235"
server_key_iv: "721be33d97fbf2b3d49f494b2ab6b8bd"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.805761 24763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:30:49.806411 24778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.806572 24763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:49.806667 24763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "52e1f54f5c4a456f9904452e3ccf4b15"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "e7ae3cde91f0079c76939217be019235"
server_key_iv: "721be33d97fbf2b3d49f494b2ab6b8bd"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.806746 24763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:49.814646 24763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:49.816308 24763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:49.816452 24763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:49.816650 24763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:49.816917 24763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:49.816967 24763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.817004 24763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:49.817030 24763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.822211 24763 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:36045
I20260929 03:30:49.822305 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:36045 every 8 connection(s)
I20260929 03:30:49.822551 24763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:49.824669 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24763
I20260929 03:30:49.824770 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:49.824977 6610 external_mini_cluster.cc:1428] Setting key cd8416f4bbda2db65cb9b83d942bb81f
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:49.832680 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:49.823225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:46885 (local address 127.6.116.190:34619)
0929 03:30:49.823289 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:49.823290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:49.823452 (+ 162us) server_negotiation.cc:407] Connection header received
0929 03:30:49.823878 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:49.823878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:49.823889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:49.823935 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:49.824897 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.825575 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.827130 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.827486 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.827561 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:49.829806 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:49.829808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:49.829808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:49.829817 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:49.831318 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.831740 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.831741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.831742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.831798 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.832057 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.832058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.832059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.832168 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:49.832236 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:49.832495 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 03:30:49.832580 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:30:49.832849 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:49.823210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34619 (local address 127.6.116.129:46885)
0929 03:30:49.823355 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:49.823391 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:49.823804 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:49.823989 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:49.823992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:49.824552 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:49.824816 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:49.824821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.825669 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.825670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:49.827019 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:49.827022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.827571 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.827571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:49.827667 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:49.828148 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:49.828157 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:49.829708 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:49.831425 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:49.831431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:49.831433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:49.831647 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:49.831862 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:49.831863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:49.831863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:49.831976 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:49.832243 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:49.832250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:49.832373 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 03:30:49.832473 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:30:49.832619 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
I20260929 03:30:49.833339 24892 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34619
I20260929 03:30:49.833434 24892 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:49.833603 24892 heartbeater.cc:511] Master 127.6.116.190:34619 requested a full tablet report, sending...
I20260929 03:30:49.834052 24701 ts_manager.cc:194] Registered new tserver with Master: 52e1f54f5c4a456f9904452e3ccf4b15 (127.6.116.129:36045)
I20260929 03:30:49.834628 24701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:46885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:30:49.847631 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:49.922657 24897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:49.922814 24897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:49.922866 24897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:49.923918 24897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:49.923955 24897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:49.923979 24897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:49.925222 24897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:49.925381 24897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:49.925526 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:49.930083 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:49.930095 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:49.930078 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:49.930295 24897 server_base.cc:1034] running on GCE node
I20260929 03:30:49.930577 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:49.930827 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:49.931980 24897 hybrid_clock.cc:648] HybridClock initialized: now 1790652649931951 us; error 57 us; skew 500 ppm
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:49.934175 24897 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:30:49.934585 24897 webserver.cc:460] Webserver started at http://127.6.116.130:38403/ using document root <none> and password file <none>
I20260929 03:30:49.934809 24897 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:49.934873 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:49.935003 24897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:49.935915 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "edeaa99e9f5a4d089366a0e99b163180"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "978d45042c25c5ed5e80a40b4bd1b93c"
server_key_iv: "6b17befc18ae9d1946d1ce60e0307781"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.936240 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "edeaa99e9f5a4d089366a0e99b163180"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "978d45042c25c5ed5e80a40b4bd1b93c"
server_key_iv: "6b17befc18ae9d1946d1ce60e0307781"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.937425 24897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:49.938089 24912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.938252 24897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:49.938339 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "edeaa99e9f5a4d089366a0e99b163180"
format_stamp: "Formatted at 2026-09-29 03:30:49 on dist-test-slave-z2zg"
server_key: "978d45042c25c5ed5e80a40b4bd1b93c"
server_key_iv: "6b17befc18ae9d1946d1ce60e0307781"
server_key_version: "encryptionkey@0"
I20260929 03:30:49.938419 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:49.953130 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:49.954751 24897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:49.954895 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:49.955137 24897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:49.955410 24897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:49.955459 24897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.955497 24897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:49.955523 24897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:49.960824 24897 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:45209
I20260929 03:30:49.960875 25025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:45209 every 8 connection(s)
I20260929 03:30:49.961186 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:49.961462 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 24897
I20260929 03:30:49.961571 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:49.961768 6610 external_mini_cluster.cc:1428] Setting key bda76f2e060fefc774aa8e2161fb9316
Sep 29 03:30:49 dist-test-slave-z2zg krb5kdc[24656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652649, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
I20260929 03:30:49.970518 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:49.961879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:53195 (local address 127.6.116.190:34619)
0929 03:30:49.961944 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:49.961946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:49.962116 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:30:49.962610 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:49.962611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:49.962622 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:49.962687 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:49.963498 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.964112 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.965229 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.965429 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.965507 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:49.967843 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:49.967845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:49.967845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:49.967851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:49.969143 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.969622 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.969623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.969623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.969672 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:49.969927 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:49.969927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:49.969928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:49.970016 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:49.970062 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:49.970384 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 03:30:49.970443 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:30:49.970762 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:49.961840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34619 (local address 127.6.116.130:53195)
0929 03:30:49.961996 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:49.962037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:49.962498 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:49.962730 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:49.962733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:49.963247 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:49.963399 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:49.963403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.964198 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.964199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:49.965146 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:49.965148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:49.965515 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:49.965516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:49.965645 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:49.966082 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:49.966090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:49.967757 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:49.969225 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:49.969232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:49.969234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:49.969531 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:49.969736 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:49.969737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:49.969737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:49.969866 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:49.970078 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:49.970084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:49.970242 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 03:30:49.970371 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 03:30:49.970544 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260929 03:30:49.971276 25026 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34619
I20260929 03:30:49.971361 25026 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:49.971475 25026 heartbeater.cc:511] Master 127.6.116.190:34619 requested a full tablet report, sending...
I20260929 03:30:49.971788 24701 ts_manager.cc:194] Registered new tserver with Master: edeaa99e9f5a4d089366a0e99b163180 (127.6.116.130:45209)
I20260929 03:30:49.972224 24701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:53195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:30:49.986171 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:50.062242 25031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:50.062430 25031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:50.062453 25031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:50.063534 25031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:50.063565 25031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:50.063592 25031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:50.064920 25031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:34619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:50.065076 25031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:50.065228 25031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:50.069948 25039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.070022 25031 server_base.cc:1034] running on GCE node
W20260929 03:30:50.069948 25037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.069936 25036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.070415 25031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:50.070664 25031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:50.071823 25031 hybrid_clock.cc:648] HybridClock initialized: now 1790652650071811 us; error 26 us; skew 500 ppm
Sep 29 03:30:50 dist-test-slave-z2zg krb5kdc[24656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:50.074151 25031 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:30:50.074585 25031 webserver.cc:460] Webserver started at http://127.6.116.131:42225/ using document root <none> and password file <none>
I20260929 03:30:50.074790 25031 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:50.074852 25031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:50.074978 25031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:50.075896 25031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "2535a6887c364732affa6438aecf7bfe"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "d50b5ca4f49987710e3e6425a854d08b"
server_key_iv: "09c8cdb2cb97c88f748ceb8d30d7817d"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.076216 25031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "2535a6887c364732affa6438aecf7bfe"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "d50b5ca4f49987710e3e6425a854d08b"
server_key_iv: "09c8cdb2cb97c88f748ceb8d30d7817d"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.077360 25031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:50.078082 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.078259 25031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:50.078348 25031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "2535a6887c364732affa6438aecf7bfe"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "d50b5ca4f49987710e3e6425a854d08b"
server_key_iv: "09c8cdb2cb97c88f748ceb8d30d7817d"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.078428 25031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:50.092800 25031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:50.094399 25031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:50.094551 25031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:50.094758 25031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:50.095027 25031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:50.095125 25031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.095162 25031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:50.095192 25031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.100533 25031 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:35191
I20260929 03:30:50.100566 25159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:35191 every 8 connection(s)
I20260929 03:30:50.100828 25031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
Sep 29 03:30:50 dist-test-slave-z2zg krb5kdc[24656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:50.109730 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.101456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:58873 (local address 127.6.116.190:34619)
0929 03:30:50.101500 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.101502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.101676 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:30:50.102142 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.102143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.102156 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.102221 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.103181 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.103697 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.104896 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.105087 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.105164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.107230 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.107231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.107232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:50.107239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.108374 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:50.108752 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:50.108754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:50.108754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:50.108812 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:50.109044 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:50.109046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:50.109046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:50.109155 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.109227 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.109548 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.109645 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:30:50.109853 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.101422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:34619 (local address 127.6.116.131:58873)
0929 03:30:50.101558 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.101599 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.102040 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.102279 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.102282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.102812 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:50.103088 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.103095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.103789 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.103790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.104767 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.104770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.105171 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.105172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.105290 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.105717 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:50.105734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:50.107142 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:50.108466 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:50.108472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:50.108474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:50.108677 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:50.108879 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:50.108880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:50.108880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:50.108977 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:50.109229 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:50.109236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:50.109418 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.109513 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.109653 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260929 03:30:50.110379 25160 heartbeater.cc:347] Connected to a master server at 127.6.116.190:34619
I20260929 03:30:50.110406 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25031
I20260929 03:30:50.110476 25160 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:50.110502 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:50.110648 25160 heartbeater.cc:511] Master 127.6.116.190:34619 requested a full tablet report, sending...
I20260929 03:30:50.110702 6610 external_mini_cluster.cc:1428] Setting key ff21768edeb3ad5b24144e0f827efaa1
I20260929 03:30:50.111042 24701 ts_manager.cc:194] Registered new tserver with Master: 2535a6887c364732affa6438aecf7bfe (127.6.116.131:35191)
I20260929 03:30:50.111567 24701 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:58873
I20260929 03:30:50.111672 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:50.118492 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.113149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.6.116.190:34619)
0929 03:30:50.113186 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.113189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.113327 (+ 138us) server_negotiation.cc:407] Connection header received
0929 03:30:50.113360 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.113361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.113368 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.113408 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.113953 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.114375 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.115016 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.115240 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.115289 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.116369 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.116370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.116370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:50.116376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.117397 (+ 1021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:50.117784 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:50.117784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:50.117785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:50.117828 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:50.117975 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:50.117975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:50.117975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:50.118076 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.118148 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.118307 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.118350 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:30:50.119926 24701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:50.121382 24701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:50.127241 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45209 (local address 127.0.0.1:44786)
0929 03:30:50.123582 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.123618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.123681 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.124037 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.124039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.124041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.124102 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.124104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.125085 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.125086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.126048 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.126051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.126940 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.126941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.127015 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.127028 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.127123 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.127171 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
I20260929 03:30:50.127264 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.6.116.130:45209)
0929 03:30:50.123783 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.123787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.123798 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:30:50.123852 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.123854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.123868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.123942 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.124237 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.124972 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.126166 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.126853 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.126934 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.126993 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.127137 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.127179 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":95,"threads_started":1}
I20260929 03:30:50.127688 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:35191 (local address 127.0.0.1:58758)
0929 03:30:50.123787 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.123799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.123852 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.124271 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.124272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.124274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.124317 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.124319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.125583 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.125584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.126576 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.126578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.127445 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.127447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.127527 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.127532 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.127571 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.127624 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":56,"threads_started":1}
I20260929 03:30:50.128019 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.6.116.131:35191)
0929 03:30:50.124093 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.124098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.124108 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:30:50.124144 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.124146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.124161 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.124395 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.124450 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.125482 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.126669 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.127341 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.127844 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.127899 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.127921 (+ 22us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.127953 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":88,"threads_started":1}
I20260929 03:30:50.128055 24960 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad81838379e4135a8dfe21d85a53982 (DEFAULT_TABLE table=test-table [id=a813cda087fa4a1ca7d50aa45c3b3887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.128290 24960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad81838379e4135a8dfe21d85a53982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.128594 25094 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad81838379e4135a8dfe21d85a53982 (DEFAULT_TABLE table=test-table [id=a813cda087fa4a1ca7d50aa45c3b3887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.128880 25094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad81838379e4135a8dfe21d85a53982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.129072 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36045 (local address 127.0.0.1:39584)
0929 03:30:50.123896 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.123906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.123946 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.124332 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.124333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.124335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.124368 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.124370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.126688 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.126689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.127746 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.127750 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.128834 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.128835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.128892 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.128897 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.128961 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.129013 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260929 03:30:50.129107 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.123912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.6.116.129:36045)
0929 03:30:50.124107 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.124111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.124122 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:30:50.124158 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.124160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.124174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.124249 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.124451 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.125474 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.127817 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.128673 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.128750 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.128798 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.128993 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.129032 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260929 03:30:50.129701 24826 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad81838379e4135a8dfe21d85a53982 (DEFAULT_TABLE table=test-table [id=a813cda087fa4a1ca7d50aa45c3b3887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.130052 24826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad81838379e4135a8dfe21d85a53982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.130849 25174 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Bootstrap starting.
I20260929 03:30:50.131719 25174 tablet_bootstrap.cc:654] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.132162 25175 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Bootstrap starting.
I20260929 03:30:50.132314 25174 log.cc:824] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.132822 25175 tablet_bootstrap.cc:654] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.133225 25174 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: No bootstrap required, opened a new log
I20260929 03:30:50.133299 25174 ts_tablet_manager.cc:1397] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:50.133354 25175 log.cc:824] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.133569 25177 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: Bootstrap starting.
I20260929 03:30:50.134096 25175 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: No bootstrap required, opened a new log
I20260929 03:30:50.134163 25175 ts_tablet_manager.cc:1397] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:50.134191 25177 tablet_bootstrap.cc:654] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.134655 25177 log.cc:824] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.135376 25177 tablet_bootstrap.cc:492] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: No bootstrap required, opened a new log
I20260929 03:30:50.135456 25177 ts_tablet_manager.cc:1397] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:50.135566 25174 raft_consensus.cc:348] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.135740 25174 raft_consensus.cc:374] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.135775 25174 raft_consensus.cc:729] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edeaa99e9f5a4d089366a0e99b163180, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.135900 25174 consensus_queue.cc:260] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.136178 25174 ts_tablet_manager.cc:1428] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:50.136214 25026 heartbeater.cc:503] Master 127.6.116.190:34619 was elected leader, sending a full tablet report...
I20260929 03:30:50.136289 25175 raft_consensus.cc:348] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.136408 25175 raft_consensus.cc:374] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.136468 25175 raft_consensus.cc:729] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2535a6887c364732affa6438aecf7bfe, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.136615 25175 consensus_queue.cc:260] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.136888 25160 heartbeater.cc:503] Master 127.6.116.190:34619 was elected leader, sending a full tablet report...
I20260929 03:30:50.136870 25175 ts_tablet_manager.cc:1428] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:50.137437 25177 raft_consensus.cc:348] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.137640 25177 raft_consensus.cc:374] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.137691 25177 raft_consensus.cc:729] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e1f54f5c4a456f9904452e3ccf4b15, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.137802 25177 consensus_queue.cc:260] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.138058 25177 ts_tablet_manager.cc:1428] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:50.138123 24892 heartbeater.cc:503] Master 127.6.116.190:34619 was elected leader, sending a full tablet report...
I20260929 03:30:50.188053 25181 raft_consensus.cc:482] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:50.188138 25181 raft_consensus.cc:504] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.188369 25181 leader_election.cc:288] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edeaa99e9f5a4d089366a0e99b163180 (127.6.116.130:45209), 52e1f54f5c4a456f9904452e3ccf4b15 (127.6.116.129:36045)
I20260929 03:30:50.191942 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.188619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:44805 (local address 127.6.116.130:45209)
0929 03:30:50.188652 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.188653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.188667 (+ 14us) server_negotiation.cc:407] Connection header received
0929 03:30:50.188688 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.188688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.188695 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.188749 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.188909 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.189636 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.190999 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.191594 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.191671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.191679 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.191846 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.191882 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 03:30:50.191942 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.188573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:55507 (local address 127.6.116.129:36045)
0929 03:30:50.188615 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.188617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.188797 (+ 180us) server_negotiation.cc:407] Connection header received
0929 03:30:50.188829 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.188830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.188837 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.188901 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.189018 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.189671 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.190992 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.191598 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.191671 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.191679 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.191846 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.191880 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 03:30:50.191970 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.188515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:45209 (local address 127.6.116.131:44805)
0929 03:30:50.188550 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.188560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.188598 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.188771 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.188772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.188774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.188836 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.188838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.189735 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.189736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.190880 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.190882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.191683 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.191684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.191765 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.191767 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.191828 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.191881 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":48}
I20260929 03:30:50.191983 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.188517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36045 (local address 127.6.116.131:55507)
0929 03:30:50.188699 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.188707 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.188777 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.188902 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.188903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.188904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.188957 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.188958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.189748 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.189748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.190880 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.190882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.191683 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.191684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.191762 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.191767 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.191828 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.191880 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":129,"thread_start_us":85,"threads_started":1}
I20260929 03:30:50.192312 24846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad81838379e4135a8dfe21d85a53982" candidate_uuid: "2535a6887c364732affa6438aecf7bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" is_pre_election: true
I20260929 03:30:50.192322 24980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad81838379e4135a8dfe21d85a53982" candidate_uuid: "2535a6887c364732affa6438aecf7bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edeaa99e9f5a4d089366a0e99b163180" is_pre_election: true
I20260929 03:30:50.192437 24980 raft_consensus.cc:2454] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2535a6887c364732affa6438aecf7bfe in term 0.
I20260929 03:30:50.192437 24846 raft_consensus.cc:2454] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2535a6887c364732affa6438aecf7bfe in term 0.
I20260929 03:30:50.192657 25048 leader_election.cc:302] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2535a6887c364732affa6438aecf7bfe, 52e1f54f5c4a456f9904452e3ccf4b15; no voters:
I20260929 03:30:50.192832 25181 raft_consensus.cc:2781] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:50.192886 25181 raft_consensus.cc:482] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:50.192901 25181 raft_consensus.cc:3037] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.193441 25181 raft_consensus.cc:504] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.193568 25181 leader_election.cc:288] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [CANDIDATE]: Term 1 election: Requested vote from peers edeaa99e9f5a4d089366a0e99b163180 (127.6.116.130:45209), 52e1f54f5c4a456f9904452e3ccf4b15 (127.6.116.129:36045)
I20260929 03:30:50.193739 24846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad81838379e4135a8dfe21d85a53982" candidate_uuid: "2535a6887c364732affa6438aecf7bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52e1f54f5c4a456f9904452e3ccf4b15"
I20260929 03:30:50.193778 24980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad81838379e4135a8dfe21d85a53982" candidate_uuid: "2535a6887c364732affa6438aecf7bfe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edeaa99e9f5a4d089366a0e99b163180"
I20260929 03:30:50.193853 24846 raft_consensus.cc:3037] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.193849 24980 raft_consensus.cc:3037] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.194474 24980 raft_consensus.cc:2454] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2535a6887c364732affa6438aecf7bfe in term 1.
I20260929 03:30:50.194485 24846 raft_consensus.cc:2454] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2535a6887c364732affa6438aecf7bfe in term 1.
I20260929 03:30:50.194643 25050 leader_election.cc:302] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2535a6887c364732affa6438aecf7bfe, edeaa99e9f5a4d089366a0e99b163180; no voters:
I20260929 03:30:50.194761 25181 raft_consensus.cc:2781] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:50.194895 25181 raft_consensus.cc:686] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 1 LEADER]: Becoming Leader. State: Replica: 2535a6887c364732affa6438aecf7bfe, State: Running, Role: LEADER
I20260929 03:30:50.194977 25181 consensus_queue.cc:237] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } }
I20260929 03:30:50.195726 24701 catalog_manager.cc:5495] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe reported cstate change: term changed from 0 to 1, leader changed from <none> to 2535a6887c364732affa6438aecf7bfe (127.6.116.131). New cstate: current_term: 1 leader_uuid: "2535a6887c364732affa6438aecf7bfe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2535a6887c364732affa6438aecf7bfe" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 35191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 } health_report { overall_health: UNKNOWN } } }
W20260929 03:30:50.211699 25027 tablet.cc:2369] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:50.218436 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.216481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.6.116.131:35191)
0929 03:30:50.216521 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.216522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.216528 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:30:50.216549 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.216549 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.216562 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.216608 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:50.216738 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.217143 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.217808 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.217969 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.218012 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.218142 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:50.218213 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:50.218273 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.218316 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.218323 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:30:50.219915 24980 raft_consensus.cc:1261] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 1 FOLLOWER]: Refusing update from remote peer 2535a6887c364732affa6438aecf7bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:50.219939 24846 raft_consensus.cc:1261] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 1 FOLLOWER]: Refusing update from remote peer 2535a6887c364732affa6438aecf7bfe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:50.220225 25184 consensus_queue.cc:1035] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "edeaa99e9f5a4d089366a0e99b163180" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:50.220367 25181 consensus_queue.cc:1035] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [LEADER]: Connected to new peer: Peer: permanent_uuid: "52e1f54f5c4a456f9904452e3ccf4b15" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 36045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:50.222285 25187 mvcc.cc:205] Tried to move back new op lower bound from 7334513255298637824 to 7334513255199363072. Current Snapshot: MvccSnapshot[applied={T|T < 7334513255298637824}]
I20260929 03:30:50.222414 25188 mvcc.cc:205] Tried to move back new op lower bound from 7334513255298637824 to 7334513255199363072. Current Snapshot: MvccSnapshot[applied={T|T < 7334513255298637824}]
I20260929 03:30:50.223171 25189 mvcc.cc:205] Tried to move back new op lower bound from 7334513255298637824 to 7334513255199363072. Current Snapshot: MvccSnapshot[applied={T|T < 7334513255298637824}]
I20260929 03:30:50.224527 24701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:50.224614 24701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:50.225281 24701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b075ebb347b4088b93b566c7c24dd34: Sending DeleteTablet for 3 replicas of tablet 9ad81838379e4135a8dfe21d85a53982
I20260929 03:30:50.225606 25094 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad81838379e4135a8dfe21d85a53982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:50 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:58758
I20260929 03:30:50.225626 24960 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad81838379e4135a8dfe21d85a53982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:50 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:44786
I20260929 03:30:50.225677 24826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad81838379e4135a8dfe21d85a53982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:50 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:39584
I20260929 03:30:50.225852 25197 tablet_replica.cc:330] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: stopping tablet replica
I20260929 03:30:50.225867 25199 tablet_replica.cc:330] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15: stopping tablet replica
I20260929 03:30:50.225947 25199 raft_consensus.cc:2229] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:50.225947 25197 raft_consensus.cc:2229] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:50.226045 25199 raft_consensus.cc:2258] T 9ad81838379e4135a8dfe21d85a53982 P 52e1f54f5c4a456f9904452e3ccf4b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:50.226037 25197 raft_consensus.cc:2258] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:50.226061 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24763
I20260929 03:30:50.226872 25197 ts_tablet_manager.cc:1905] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:50.228741 25197 ts_tablet_manager.cc:1918] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:50.228816 25197 log.cc:1196] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/9ad81838379e4135a8dfe21d85a53982
I20260929 03:30:50.229082 25197 ts_tablet_manager.cc:1939] T 9ad81838379e4135a8dfe21d85a53982 P 2535a6887c364732affa6438aecf7bfe: Deleting consensus metadata
I20260929 03:30:50.229430 25198 tablet_replica.cc:330] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: stopping tablet replica
I20260929 03:30:50.229467 24686 catalog_manager.cc:4841] TS 2535a6887c364732affa6438aecf7bfe (127.6.116.131:35191): tablet 9ad81838379e4135a8dfe21d85a53982 (table test-table [id=a813cda087fa4a1ca7d50aa45c3b3887]) successfully deleted
I20260929 03:30:50.229518 25198 raft_consensus.cc:2229] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:50.229602 25198 raft_consensus.cc:2258] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:50.229967 25198 ts_tablet_manager.cc:1905] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:50.231804 25198 ts_tablet_manager.cc:1918] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:50.231875 25198 log.cc:1196] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/9ad81838379e4135a8dfe21d85a53982
I20260929 03:30:50.232134 25198 ts_tablet_manager.cc:1939] T 9ad81838379e4135a8dfe21d85a53982 P edeaa99e9f5a4d089366a0e99b163180: Deleting consensus metadata
I20260929 03:30:50.232517 24688 catalog_manager.cc:4841] TS edeaa99e9f5a4d089366a0e99b163180 (127.6.116.130:45209): tablet 9ad81838379e4135a8dfe21d85a53982 (table test-table [id=a813cda087fa4a1ca7d50aa45c3b3887]) successfully deleted
W20260929 03:30:50.232743 24686 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:36045 (error 108)
I20260929 03:30:50.233032 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24897
I20260929 03:30:50.236909 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.236812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:36045 (local address 127.0.0.1:39594)
0929 03:30:50.236851 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.236878 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:36045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 03:30:50.237002 24686 catalog_manager.cc:4584] TS 52e1f54f5c4a456f9904452e3ccf4b15 (127.6.116.129:36045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ad81838379e4135a8dfe21d85a53982: Network error: Client connection negotiation failed: client connection to 127.6.116.129:36045: connect: Connection refused (error 111)
I20260929 03:30:50.238096 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25031
I20260929 03:30:50.242776 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 24669
2026-09-29T03:30:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1840 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T03:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:50Z Disabled control of system clock
I20260929 03:30:50.268944 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:35487
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:35487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:50.348784 25204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:50.348974 25204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:50.348994 25204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:50.349999 25204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:50.350031 25204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:50.350044 25204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:50.350061 25204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:50.350086 25204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:50.351315 25204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:35487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:35487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:50.351480 25204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:50.351675 25204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:50.356066 25210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.356086 25209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.356107 25212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.356170 25204 server_base.cc:1034] running on GCE node
I20260929 03:30:50.356372 25204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:50.356585 25204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:50.357725 25204 hybrid_clock.cc:648] HybridClock initialized: now 1790652650357710 us; error 29 us; skew 500 ppm
I20260929 03:30:50.358856 25204 webserver.cc:460] Webserver started at http://127.6.116.190:38983/ using document root <none> and password file <none>
I20260929 03:30:50.359030 25204 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:50.359177 25204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:50.359313 25204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:50.360689 25204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "fc44ac5c29104bccb9aab50bdd2ee942"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "06b22020131e7adf99dc842c06a7a77e"
server_key_iv: "a7797f656da59937d9ff8440aba0e502"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.361074 25204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "fc44ac5c29104bccb9aab50bdd2ee942"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "06b22020131e7adf99dc842c06a7a77e"
server_key_iv: "a7797f656da59937d9ff8440aba0e502"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.362546 25204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:30:50.363299 25218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.363461 25204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:50.363554 25204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "fc44ac5c29104bccb9aab50bdd2ee942"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "06b22020131e7adf99dc842c06a7a77e"
server_key_iv: "a7797f656da59937d9ff8440aba0e502"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.363633 25204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:50.374475 25204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:50.374747 25204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:50.374892 25204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:50.378695 25204 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:35487
I20260929 03:30:50.378723 25270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:35487 every 8 connection(s)
I20260929 03:30:50.379032 25204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:50.379917 25271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.381927 25271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: Bootstrap starting.
I20260929 03:30:50.382471 25271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.382660 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25204
I20260929 03:30:50.382791 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:50.382786 25271 log.cc:824] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.382982 6610 external_mini_cluster.cc:1428] Setting key 2c980a0a393450f5b3f6ae062c8d8d54
I20260929 03:30:50.383507 25271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: No bootstrap required, opened a new log
I20260929 03:30:50.386171 25271 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } }
I20260929 03:30:50.386323 25271 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.386353 25271 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc44ac5c29104bccb9aab50bdd2ee942, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.386453 25271 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } }
I20260929 03:30:50.386515 25271 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:50.386543 25271 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:50.386596 25271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.387220 25271 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } }
I20260929 03:30:50.387322 25271 leader_election.cc:302] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc44ac5c29104bccb9aab50bdd2ee942; no voters:
I20260929 03:30:50.387526 25271 leader_election.cc:288] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:50.387619 25276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:50.387779 25271 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:50.387806 25276 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [term 1 LEADER]: Becoming Leader. State: Replica: fc44ac5c29104bccb9aab50bdd2ee942, State: Running, Role: LEADER
I20260929 03:30:50.387979 25276 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } }
I20260929 03:30:50.388474 25278 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc44ac5c29104bccb9aab50bdd2ee942. Latest consensus state: current_term: 1 leader_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } } }
I20260929 03:30:50.388551 25278 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:50.388768 25277 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc44ac5c29104bccb9aab50bdd2ee942" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 35487 } } }
I20260929 03:30:50.388839 25277 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:50.388996 25285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:50.389456 25285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:50.391000 25285 catalog_manager.cc:1347] Generated new cluster ID: e29041ac5874414fab411721c8a75d5f
I20260929 03:30:50.391103 25285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:50.391510 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.383472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.6.116.190:35487)
0929 03:30:50.383701 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.383709 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.383739 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:30:50.384686 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.384695 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.384767 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.384919 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.385096 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.386536 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.387593 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.388189 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.388642 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.390646 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.390650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.390650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:50.390863 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.390901 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:50.390916 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.391203 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.391235 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.391309 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260929 03:30:50.411422 25285 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:50.411863 25285 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:50.413986 25285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: Generated new TSK 0
I20260929 03:30:50.414131 25285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:50.418471 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--builtin_ntp_servers=127.6.116.148:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:50.497978 25295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:50.498173 25295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:50.498195 25295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:50.499344 25295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:50.499382 25295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:50.499405 25295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:50.500667 25295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:50.500896 25295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:50.501209 25295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:50.506744 25300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.506795 25301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.507086 25295 server_base.cc:1034] running on GCE node
W20260929 03:30:50.507179 25303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.507385 25295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:50.507647 25295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:50.508801 25295 hybrid_clock.cc:648] HybridClock initialized: now 1790652650508776 us; error 37 us; skew 500 ppm
I20260929 03:30:50.509967 25295 webserver.cc:460] Webserver started at http://127.6.116.129:46437/ using document root <none> and password file <none>
I20260929 03:30:50.510201 25295 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:50.510288 25295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:50.510435 25295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:50.511555 25295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "9b6d26f70bb74a7496d9506e66709bf6"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "5a3aa0efc5f051cb3b456f964dbc6cb7"
server_key_iv: "63881d3fc130eaef84fae61677953b68"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.511914 25295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "9b6d26f70bb74a7496d9506e66709bf6"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "5a3aa0efc5f051cb3b456f964dbc6cb7"
server_key_iv: "63881d3fc130eaef84fae61677953b68"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.513147 25295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:50.513823 25309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.514003 25295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:50.514099 25295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "9b6d26f70bb74a7496d9506e66709bf6"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "5a3aa0efc5f051cb3b456f964dbc6cb7"
server_key_iv: "63881d3fc130eaef84fae61677953b68"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.514181 25295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:50.534129 25295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:50.534441 25295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:50.534611 25295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:50.534835 25295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:50.535173 25295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:50.535228 25295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.535262 25295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:50.535290 25295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.540792 25295 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:44013
I20260929 03:30:50.540843 25422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:44013 every 8 connection(s)
I20260929 03:30:50.541128 25295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:50.545729 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25295
I20260929 03:30:50.545812 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:50.546012 6610 external_mini_cluster.cc:1428] Setting key 70108ac5efda7be1116f45bc6796469d
I20260929 03:30:50.546265 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.541789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:35333 (local address 127.6.116.190:35487)
0929 03:30:50.541850 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.541852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.542086 (+ 234us) server_negotiation.cc:407] Connection header received
0929 03:30:50.542540 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.542541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.542552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.542605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.543258 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.543743 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.544754 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.544942 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.545002 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.545949 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.545950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.545951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:50.545956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.545980 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:50.545985 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.546065 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.546175 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.546203 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:30:50.546516 25425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.541777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:35487 (local address 127.6.116.129:35333)
0929 03:30:50.541965 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.542002 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.542448 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.542664 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.542667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.542675 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:50.543171 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.543176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.543858 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.543859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.544661 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.544664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.545038 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.545039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.545150 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.545788 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:50.545789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:50.545829 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:50.545859 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:50.546069 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:50.546075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:50.546078 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.546153 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.546281 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260929 03:30:50.547084 25423 heartbeater.cc:347] Connected to a master server at 127.6.116.190:35487
I20260929 03:30:50.547179 25423 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:50.547226 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--builtin_ntp_servers=127.6.116.148:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:30:50.547366 25423 heartbeater.cc:511] Master 127.6.116.190:35487 requested a full tablet report, sending...
I20260929 03:30:50.547811 25235 ts_manager.cc:194] Registered new tserver with Master: 9b6d26f70bb74a7496d9506e66709bf6 (127.6.116.129:44013)
I20260929 03:30:50.548385 25235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.129:35333
W20260929 03:30:50.627398 25426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:50.627614 25426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:50.627647 25426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:50.628726 25426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:50.628787 25426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:50.628827 25426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:50.630123 25426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:50.630371 25426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:50.630640 25426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:50.635309 25434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.635329 25431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.635314 25432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.635547 25426 server_base.cc:1034] running on GCE node
I20260929 03:30:50.635713 25426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:50.635897 25426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:50.637044 25426 hybrid_clock.cc:648] HybridClock initialized: now 1790652650637016 us; error 29 us; skew 500 ppm
I20260929 03:30:50.638151 25426 webserver.cc:460] Webserver started at http://127.6.116.130:46469/ using document root <none> and password file <none>
I20260929 03:30:50.638374 25426 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:50.638418 25426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:50.638561 25426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:50.639580 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "6ba4fa65a8c44196bf96d581011b61f3"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "98c4960573f16ab45982dd7c9209cb07"
server_key_iv: "b93c28f03583fab1d7c302156a9693d7"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.639920 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "6ba4fa65a8c44196bf96d581011b61f3"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "98c4960573f16ab45982dd7c9209cb07"
server_key_iv: "b93c28f03583fab1d7c302156a9693d7"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.641146 25426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:50.641827 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.641985 25426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:50.642091 25426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "6ba4fa65a8c44196bf96d581011b61f3"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "98c4960573f16ab45982dd7c9209cb07"
server_key_iv: "b93c28f03583fab1d7c302156a9693d7"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.642165 25426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:50.654084 25426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:50.654325 25426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:50.654456 25426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:50.654650 25426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:50.654944 25426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:50.654997 25426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.655059 25426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:50.655087 25426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.660378 25426 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:41263
I20260929 03:30:50.660444 25553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:41263 every 8 connection(s)
I20260929 03:30:50.660727 25426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:50.661772 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25426
I20260929 03:30:50.661851 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:50.662048 6610 external_mini_cluster.cc:1428] Setting key b2eebc2f59db409e73a8f756b823e12d
I20260929 03:30:50.662880 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--builtin_ntp_servers=127.6.116.148:34907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:30:50.666355 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.661469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:56115 (local address 127.6.116.190:35487)
0929 03:30:50.661511 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.661513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.661708 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:30:50.662318 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.662320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.662332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.662391 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.663107 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.663582 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.664916 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.665093 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.665153 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.666062 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.666063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.666064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:50.666067 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.666081 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:50.666084 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.666151 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.666271 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.666299 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:30:50.666620 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.661441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:35487 (local address 127.6.116.130:56115)
0929 03:30:50.661600 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.661636 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.662219 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.662453 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.662457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.662465 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:50.662912 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.662937 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.663668 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.663669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.664826 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.664829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.665190 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.665191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.665362 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.665910 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:50.665910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:50.665947 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:50.665974 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:50.666133 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:50.666137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:50.666141 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.666256 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.666368 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":84,"threads_started":1}
I20260929 03:30:50.667060 25554 heartbeater.cc:347] Connected to a master server at 127.6.116.190:35487
I20260929 03:30:50.667146 25554 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:50.667339 25554 heartbeater.cc:511] Master 127.6.116.190:35487 requested a full tablet report, sending...
I20260929 03:30:50.667701 25235 ts_manager.cc:194] Registered new tserver with Master: 6ba4fa65a8c44196bf96d581011b61f3 (127.6.116.130:41263)
I20260929 03:30:50.668072 25235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.130:56115
W20260929 03:30:50.741421 25557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:50.741643 25557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:50.741677 25557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:50.742790 25557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:50.742857 25557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:50.742897 25557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:50.744172 25557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:34907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:35487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:50.744418 25557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:50.744714 25557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:50.749342 25562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.749320 25563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:50.749331 25565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:50.749437 25557 server_base.cc:1034] running on GCE node
I20260929 03:30:50.749677 25557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:50.749887 25557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:50.751020 25557 hybrid_clock.cc:648] HybridClock initialized: now 1790652650751001 us; error 35 us; skew 500 ppm
I20260929 03:30:50.752194 25557 webserver.cc:460] Webserver started at http://127.6.116.131:39603/ using document root <none> and password file <none>
I20260929 03:30:50.752418 25557 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:50.752460 25557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:50.752621 25557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:50.753707 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "b85d3d759eb24b8ba10ec773023874be"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "daa39c0207699582cc7867e9e8134991"
server_key_iv: "be9ee2fcb99c15f631b85a9fca1d0edf"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.754029 25557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "b85d3d759eb24b8ba10ec773023874be"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "daa39c0207699582cc7867e9e8134991"
server_key_iv: "be9ee2fcb99c15f631b85a9fca1d0edf"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.755313 25557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:50.755977 25571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.756145 25557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.756208 25557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "b85d3d759eb24b8ba10ec773023874be"
format_stamp: "Formatted at 2026-09-29 03:30:50 on dist-test-slave-z2zg"
server_key: "daa39c0207699582cc7867e9e8134991"
server_key_iv: "be9ee2fcb99c15f631b85a9fca1d0edf"
server_key_version: "encryptionkey@0"
I20260929 03:30:50.756292 25557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:50.767395 25557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:50.767642 25557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:50.767776 25557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:50.767974 25557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:50.768255 25557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:50.768285 25557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.768335 25557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:50.768363 25557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:50.773942 25557 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:39179
I20260929 03:30:50.773990 25684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:39179 every 8 connection(s)
I20260929 03:30:50.774266 25557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:30:50.777628 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25557
I20260929 03:30:50.777745 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:50.777959 6610 external_mini_cluster.cc:1428] Setting key f089b6282d43bfa8e6524dc3c23963bb
I20260929 03:30:50.782779 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.774896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:52243 (local address 127.6.116.190:35487)
0929 03:30:50.774972 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.774975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.775208 (+ 233us) server_negotiation.cc:407] Connection header received
0929 03:30:50.775597 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.775598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.775606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.775653 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.776201 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.776657 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.777713 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.777961 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.778040 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.782585 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.782586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.782586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:50.782590 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.782603 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:50.782607 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.782650 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.782706 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.782725 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:30:50.782977 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.774875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:35487 (local address 127.6.116.131:52243)
0929 03:30:50.775090 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.775127 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.775524 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.775704 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.775707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.775714 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:50.776141 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.776144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.776730 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.776731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.777602 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.777604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.778062 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.778063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.778199 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.782459 (+ 4260us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:50.782459 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:50.782494 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:50.782514 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:50.782660 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:50.782663 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:50.782665 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.782707 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.782772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260929 03:30:50.783398 25685 heartbeater.cc:347] Connected to a master server at 127.6.116.190:35487
I20260929 03:30:50.783484 25685 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:50.783674 25685 heartbeater.cc:511] Master 127.6.116.190:35487 requested a full tablet report, sending...
I20260929 03:30:50.783995 25235 ts_manager.cc:194] Registered new tserver with Master: b85d3d759eb24b8ba10ec773023874be (127.6.116.131:39179)
I20260929 03:30:50.784349 25235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.131:52243
I20260929 03:30:50.789860 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:50.794824 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.791221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.6.116.190:35487)
0929 03:30:50.791272 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.791274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.791423 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:30:50.791570 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.791571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.791582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.791649 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:50.791868 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.792597 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.793491 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.793799 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.793870 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.794645 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:50.794646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:50.794647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:50.794650 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:50.794662 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:50.794665 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:50.794728 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.794769 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.794778 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:30:50.796262 25235 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:50.797708 25235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:50.803624 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.6.116.131:39179)
0929 03:30:50.799995 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.799999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.800332 (+ 333us) server_negotiation.cc:407] Connection header received
0929 03:30:50.800386 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.800388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.800401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.800466 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.800646 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801297 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.802645 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803306 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803369 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803451 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.803517 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.803548 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
I20260929 03:30:50.803619 25693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:41263 (local address 127.0.0.1:38746)
0929 03:30:50.799715 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.799750 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.799806 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.800143 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.800146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.800148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.800208 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.800209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801528 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.801548 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.802405 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.802408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803326 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.803400 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803411 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.803453 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.803509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260929 03:30:50.803805 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56318 (local address 127.6.116.129:44013)
0929 03:30:50.800115 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.800118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.800175 (+ 57us) server_negotiation.cc:407] Connection header received
0929 03:30:50.800271 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.800273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.800303 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.800393 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.800572 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801607 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.802795 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803310 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803389 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803469 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.803690 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.803729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260929 03:30:50.803872 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:39179 (local address 127.0.0.1:48088)
0929 03:30:50.800207 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.800216 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.800261 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.800516 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.800517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.800519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.800563 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.800564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801384 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.801385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.802566 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.802568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803383 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.803440 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803444 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.803505 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.803541 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":86,"threads_started":1}
I20260929 03:30:50.804015 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:44013 (local address 127.0.0.1:56318)
0929 03:30:50.800122 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.800133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.800187 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.800460 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.800461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.800463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.800499 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.800500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801681 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.801682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.802719 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.802720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803501 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803501 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.803551 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803554 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.803597 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.803632 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":74,"threads_started":1}
I20260929 03:30:50.804050 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.799757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.6.116.130:41263)
0929 03:30:50.799966 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.799970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.799982 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:30:50.800022 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.800024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.800039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.800125 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.800314 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.801434 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.802496 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.803227 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.803807 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.803903 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.803926 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.803957 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":69,"threads_started":1}
I20260929 03:30:50.804687 25619 tablet_service.cc:1467] Processing CreateTablet for tablet de379301da52494c9275df254ca3a779 (DEFAULT_TABLE table=test-table [id=ea54cbb55be046b0b2181232d4a3f110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.804687 25357 tablet_service.cc:1467] Processing CreateTablet for tablet de379301da52494c9275df254ca3a779 (DEFAULT_TABLE table=test-table [id=ea54cbb55be046b0b2181232d4a3f110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.804762 25488 tablet_service.cc:1467] Processing CreateTablet for tablet de379301da52494c9275df254ca3a779 (DEFAULT_TABLE table=test-table [id=ea54cbb55be046b0b2181232d4a3f110]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:50.805009 25619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de379301da52494c9275df254ca3a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.805009 25357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de379301da52494c9275df254ca3a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.805112 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de379301da52494c9275df254ca3a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:50.808166 25701 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: Bootstrap starting.
I20260929 03:30:50.808327 25699 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Bootstrap starting.
I20260929 03:30:50.808775 25701 tablet_bootstrap.cc:654] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.808874 25699 tablet_bootstrap.cc:654] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.809064 25701 log.cc:824] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.809181 25699 log.cc:824] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.809396 25700 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Bootstrap starting.
I20260929 03:30:50.809875 25701 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: No bootstrap required, opened a new log
I20260929 03:30:50.809948 25701 ts_tablet_manager.cc:1397] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:50.809978 25699 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: No bootstrap required, opened a new log
I20260929 03:30:50.810010 25700 tablet_bootstrap.cc:654] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:50.810030 25699 ts_tablet_manager.cc:1397] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:50.810266 25700 log.cc:824] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:50.810897 25700 tablet_bootstrap.cc:492] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: No bootstrap required, opened a new log
I20260929 03:30:50.810972 25700 ts_tablet_manager.cc:1397] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:50.812516 25699 raft_consensus.cc:348] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.812516 25701 raft_consensus.cc:348] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.812673 25701 raft_consensus.cc:374] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.812673 25699 raft_consensus.cc:374] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.812711 25701 raft_consensus.cc:729] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6d26f70bb74a7496d9506e66709bf6, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.812711 25699 raft_consensus.cc:729] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba4fa65a8c44196bf96d581011b61f3, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.812846 25701 consensus_queue.cc:260] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.812845 25699 consensus_queue.cc:260] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.812976 25700 raft_consensus.cc:348] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.813087 25701 ts_tablet_manager.cc:1428] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:50.813087 25699 ts_tablet_manager.cc:1428] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:50.813136 25700 raft_consensus.cc:374] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:50.813201 25554 heartbeater.cc:503] Master 127.6.116.190:35487 was elected leader, sending a full tablet report...
I20260929 03:30:50.813203 25700 raft_consensus.cc:729] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85d3d759eb24b8ba10ec773023874be, State: Initialized, Role: FOLLOWER
I20260929 03:30:50.813309 25700 consensus_queue.cc:260] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.813467 25423 heartbeater.cc:503] Master 127.6.116.190:35487 was elected leader, sending a full tablet report...
I20260929 03:30:50.813604 25685 heartbeater.cc:503] Master 127.6.116.190:35487 was elected leader, sending a full tablet report...
I20260929 03:30:50.813850 25700 ts_tablet_manager.cc:1428] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260929 03:30:50.911267 25555 tablet.cc:2369] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:50.974088 25707 raft_consensus.cc:482] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:50.974193 25707 raft_consensus.cc:504] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.974516 25707 leader_election.cc:288] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ba4fa65a8c44196bf96d581011b61f3 (127.6.116.130:41263), 9b6d26f70bb74a7496d9506e66709bf6 (127.6.116.129:44013)
I20260929 03:30:50.977185 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.974737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:53155 (local address 127.6.116.130:41263)
0929 03:30:50.974792 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.974794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.974803 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:50.974853 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.974854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.974869 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.974935 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.975142 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.975625 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.976534 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.976856 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.976933 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.976941 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.977089 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.977123 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 03:30:50.977185 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.974669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:41263 (local address 127.6.116.131:53155)
0929 03:30:50.974722 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.974732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.974782 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.974949 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.974950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.974951 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.975063 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.975081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.975710 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.975711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.976447 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.976449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.976932 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.976933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.977009 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.977012 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.977080 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.977121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 03:30:50.977299 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.974783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:45991 (local address 127.6.116.129:44013)
0929 03:30:50.974839 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:50.974841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:50.974969 (+ 128us) server_negotiation.cc:407] Connection header received
0929 03:30:50.975044 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:50.975045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:50.975077 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:50.975151 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:50.975299 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.975754 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.976652 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.976973 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.977052 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.977058 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:50.977207 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 03:30:50.977242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:30:50.977305 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:50.974705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:44013 (local address 127.6.116.131:45991)
0929 03:30:50.974905 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:50.974912 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:50.974973 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:50.975157 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:50.975158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:50.975160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:50.975231 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.975232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.975840 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.975841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.976564 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:50.976565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:50.977053 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:50.977054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:50.977139 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:50.977141 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:50.977202 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 03:30:50.977235 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":127,"threads_started":1}
I20260929 03:30:50.977535 25508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de379301da52494c9275df254ca3a779" candidate_uuid: "b85d3d759eb24b8ba10ec773023874be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba4fa65a8c44196bf96d581011b61f3" is_pre_election: true
I20260929 03:30:50.977703 25508 raft_consensus.cc:2454] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85d3d759eb24b8ba10ec773023874be in term 0.
I20260929 03:30:50.977676 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de379301da52494c9275df254ca3a779" candidate_uuid: "b85d3d759eb24b8ba10ec773023874be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6d26f70bb74a7496d9506e66709bf6" is_pre_election: true
I20260929 03:30:50.977828 25377 raft_consensus.cc:2454] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b85d3d759eb24b8ba10ec773023874be in term 0.
I20260929 03:30:50.977973 25574 leader_election.cc:302] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ba4fa65a8c44196bf96d581011b61f3, b85d3d759eb24b8ba10ec773023874be; no voters:
I20260929 03:30:50.978096 25707 raft_consensus.cc:2781] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:50.978174 25707 raft_consensus.cc:482] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:50.978227 25707 raft_consensus.cc:3037] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.979032 25707 raft_consensus.cc:504] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.979228 25707 leader_election.cc:288] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [CANDIDATE]: Term 1 election: Requested vote from peers 6ba4fa65a8c44196bf96d581011b61f3 (127.6.116.130:41263), 9b6d26f70bb74a7496d9506e66709bf6 (127.6.116.129:44013)
I20260929 03:30:50.979327 25508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de379301da52494c9275df254ca3a779" candidate_uuid: "b85d3d759eb24b8ba10ec773023874be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba4fa65a8c44196bf96d581011b61f3"
I20260929 03:30:50.979357 25377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de379301da52494c9275df254ca3a779" candidate_uuid: "b85d3d759eb24b8ba10ec773023874be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b6d26f70bb74a7496d9506e66709bf6"
I20260929 03:30:50.979398 25508 raft_consensus.cc:3037] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.979440 25377 raft_consensus.cc:3037] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:50.980091 25508 raft_consensus.cc:2454] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85d3d759eb24b8ba10ec773023874be in term 1.
I20260929 03:30:50.980195 25377 raft_consensus.cc:2454] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b85d3d759eb24b8ba10ec773023874be in term 1.
I20260929 03:30:50.980250 25574 leader_election.cc:302] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ba4fa65a8c44196bf96d581011b61f3, b85d3d759eb24b8ba10ec773023874be; no voters:
I20260929 03:30:50.980320 25707 raft_consensus.cc:2781] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:50.980450 25707 raft_consensus.cc:686] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 1 LEADER]: Becoming Leader. State: Replica: b85d3d759eb24b8ba10ec773023874be, State: Running, Role: LEADER
I20260929 03:30:50.980556 25707 consensus_queue.cc:237] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } }
I20260929 03:30:50.981328 25235 catalog_manager.cc:5495] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be reported cstate change: term changed from 0 to 1, leader changed from <none> to b85d3d759eb24b8ba10ec773023874be (127.6.116.131). New cstate: current_term: 1 leader_uuid: "b85d3d759eb24b8ba10ec773023874be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b85d3d759eb24b8ba10ec773023874be" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 39179 } health_report { overall_health: HEALTHY } } }
W20260929 03:30:51.024763 25686 tablet.cc:2369] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:51.028144 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:51.025771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.6.116.131:39179)
0929 03:30:51.025811 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:51.025813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:51.025821 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:51.025890 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:51.025895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:51.025912 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:51.025979 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:51.026090 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:51.026519 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:51.027392 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:51.027571 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:51.027657 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:51.027808 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:51.027898 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:51.027991 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:51.028048 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 03:30:51.028055 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:30:51.029651 25508 raft_consensus.cc:1261] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 1 FOLLOWER]: Refusing update from remote peer b85d3d759eb24b8ba10ec773023874be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:51.029968 25377 raft_consensus.cc:1261] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 1 FOLLOWER]: Refusing update from remote peer b85d3d759eb24b8ba10ec773023874be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:51.029999 25707 consensus_queue.cc:1035] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba4fa65a8c44196bf96d581011b61f3" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 41263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:51.030284 25709 consensus_queue.cc:1035] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b6d26f70bb74a7496d9506e66709bf6" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:51.034802 25234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:51.034888 25234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:51.035599 25234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc44ac5c29104bccb9aab50bdd2ee942: Sending DeleteTablet for 3 replicas of tablet de379301da52494c9275df254ca3a779
I20260929 03:30:51.035965 25488 tablet_service.cc:1514] Processing DeleteTablet for tablet de379301da52494c9275df254ca3a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:51 UTC) from {username='slave'} at 127.0.0.1:38746
I20260929 03:30:51.036001 25357 tablet_service.cc:1514] Processing DeleteTablet for tablet de379301da52494c9275df254ca3a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:51 UTC) from {username='slave'} at 127.0.0.1:56318
I20260929 03:30:51.036173 25722 tablet_replica.cc:330] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: stopping tablet replica
I20260929 03:30:51.036265 25722 raft_consensus.cc:2229] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:51.036332 25619 tablet_service.cc:1514] Processing DeleteTablet for tablet de379301da52494c9275df254ca3a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:51 UTC) from {username='slave'} at 127.0.0.1:48088
I20260929 03:30:51.036351 25723 tablet_replica.cc:330] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6: stopping tablet replica
I20260929 03:30:51.036373 25722 raft_consensus.cc:2258] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:51.036491 25723 raft_consensus.cc:2229] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:51.036625 25723 raft_consensus.cc:2258] T de379301da52494c9275df254ca3a779 P 9b6d26f70bb74a7496d9506e66709bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:51.036697 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25295
I20260929 03:30:51.038391 25722 ts_tablet_manager.cc:1905] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:51.039147 25724 tablet_replica.cc:330] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: stopping tablet replica
I20260929 03:30:51.039296 25724 raft_consensus.cc:2229] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:51.039448 25724 raft_consensus.cc:2258] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:51.039852 25724 ts_tablet_manager.cc:1905] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:51.040362 25722 ts_tablet_manager.cc:1918] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:51.040436 25722 log.cc:1196] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/de379301da52494c9275df254ca3a779
I20260929 03:30:51.040700 25722 ts_tablet_manager.cc:1939] T de379301da52494c9275df254ca3a779 P 6ba4fa65a8c44196bf96d581011b61f3: Deleting consensus metadata
I20260929 03:30:51.041066 25221 catalog_manager.cc:4841] TS 6ba4fa65a8c44196bf96d581011b61f3 (127.6.116.130:41263): tablet de379301da52494c9275df254ca3a779 (table test-table [id=ea54cbb55be046b0b2181232d4a3f110]) successfully deleted
I20260929 03:30:51.041729 25724 ts_tablet_manager.cc:1918] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:51.041851 25724 log.cc:1196] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/de379301da52494c9275df254ca3a779
I20260929 03:30:51.042193 25724 ts_tablet_manager.cc:1939] T de379301da52494c9275df254ca3a779 P b85d3d759eb24b8ba10ec773023874be: Deleting consensus metadata
W20260929 03:30:51.042223 25220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:44013 (error 108)
I20260929 03:30:51.042472 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25426
I20260929 03:30:51.042708 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:51.042525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:44013 (local address 127.0.0.1:56324)
0929 03:30:51.042631 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:51.042677 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:44013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 03:30:51.042807 25221 catalog_manager.cc:4841] TS b85d3d759eb24b8ba10ec773023874be (127.6.116.131:39179): tablet de379301da52494c9275df254ca3a779 (table test-table [id=ea54cbb55be046b0b2181232d4a3f110]) successfully deleted
W20260929 03:30:51.042986 25220 catalog_manager.cc:4584] TS 9b6d26f70bb74a7496d9506e66709bf6 (127.6.116.129:44013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de379301da52494c9275df254ca3a779: Network error: Client connection negotiation failed: client connection to 127.6.116.129:44013: connect: Connection refused (error 111)
I20260929 03:30:51.047401 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25557
I20260929 03:30:51.052139 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25204
2026-09-29T03:30:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (808 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:51 dist-test-slave-z2zg krb5kdc[25728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:51 dist-test-slave-z2zg krb5kdc[25728](info): set up 2 sockets
Sep 29 03:30:51 dist-test-slave-z2zg krb5kdc[25728](info): commencing operation
krb5kdc: starting...
W20260929 03:30:52.060400 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:30:52.165047 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:32793
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:32793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:52.240422 25741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:52.240602 25741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:52.240623 25741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:52.241943 25741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:52.241986 25741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:52.241999 25741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:52.242049 25741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:52.242069 25741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:52.243543 25741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:32793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:52.243762 25741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:52.243974 25741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:52.248262 25747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.248348 25749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.248348 25746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:52.248504 25741 server_base.cc:1034] running on GCE node
I20260929 03:30:52.248729 25741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:52.248989 25741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:52.250150 25741 hybrid_clock.cc:648] HybridClock initialized: now 1790652652250139 us; error 46 us; skew 500 ppm
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:52.252460 25741 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:30:52.252808 25741 webserver.cc:460] Webserver started at http://127.6.116.190:32821/ using document root <none> and password file <none>
I20260929 03:30:52.252967 25741 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:52.253005 25741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:52.253095 25741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:52.253960 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "f26480ecb60844d39bf1490590ab24bf"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "2c2d37b7915abc9573a84d4bea05dea8"
server_key_iv: "3d315342f65571a25f003fc6c7c5bc46"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.254240 25741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "f26480ecb60844d39bf1490590ab24bf"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "2c2d37b7915abc9573a84d4bea05dea8"
server_key_iv: "3d315342f65571a25f003fc6c7c5bc46"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.255448 25741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:52.256093 25756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.256263 25741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:52.256342 25741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "f26480ecb60844d39bf1490590ab24bf"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "2c2d37b7915abc9573a84d4bea05dea8"
server_key_iv: "3d315342f65571a25f003fc6c7c5bc46"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.256393 25741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:52.275262 25741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:52.276924 25741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:52.277024 25741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:52.280383 25741 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:32793
I20260929 03:30:52.280445 25808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:32793 every 8 connection(s)
I20260929 03:30:52.280728 25741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:52.281523 25809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:52.283453 25809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: Bootstrap starting.
I20260929 03:30:52.283962 25809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:52.284219 25809 log.cc:824] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:52.284816 25809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: No bootstrap required, opened a new log
I20260929 03:30:52.286268 25809 raft_consensus.cc:348] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } }
I20260929 03:30:52.286409 25809 raft_consensus.cc:374] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:52.286456 25809 raft_consensus.cc:729] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f26480ecb60844d39bf1490590ab24bf, State: Initialized, Role: FOLLOWER
I20260929 03:30:52.286599 25809 consensus_queue.cc:260] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } }
I20260929 03:30:52.286676 25809 raft_consensus.cc:388] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:52.286717 25809 raft_consensus.cc:482] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:52.286763 25809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:52.287313 25809 raft_consensus.cc:504] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } }
I20260929 03:30:52.287423 25809 leader_election.cc:302] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f26480ecb60844d39bf1490590ab24bf; no voters:
I20260929 03:30:52.287590 25809 leader_election.cc:288] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:52.287634 25812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:52.287808 25812 raft_consensus.cc:686] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [term 1 LEADER]: Becoming Leader. State: Replica: f26480ecb60844d39bf1490590ab24bf, State: Running, Role: LEADER
I20260929 03:30:52.287827 25809 sys_catalog.cc:564] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:52.287889 25812 consensus_queue.cc:237] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } }
I20260929 03:30:52.288283 25814 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f26480ecb60844d39bf1490590ab24bf. Latest consensus state: current_term: 1 leader_uuid: "f26480ecb60844d39bf1490590ab24bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } } }
I20260929 03:30:52.288374 25814 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:52.288386 25813 sys_catalog.cc:455] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f26480ecb60844d39bf1490590ab24bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f26480ecb60844d39bf1490590ab24bf" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 32793 } } }
I20260929 03:30:52.288439 25813 sys_catalog.cc:458] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:52.288601 25821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:52.288774 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25741
I20260929 03:30:52.288872 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:52.289108 6610 external_mini_cluster.cc:1428] Setting key 06071d9dbb7096bf59826761c02ff482
I20260929 03:30:52.289113 25821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:52.290868 25821 catalog_manager.cc:1347] Generated new cluster ID: 41acdc512d6343d5a1ca6466a9d89c96
I20260929 03:30:52.290937 25821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:52.300138 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.289655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.6.116.190:32793)
0929 03:30:52.289797 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.289805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.289834 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:30:52.289934 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.289940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.289988 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.290146 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.290862 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.292222 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.293334 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.293933 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.295654 (+ 1721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.296620 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.296624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.296624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:52.296636 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.298542 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.298917 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.298920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.298921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.299090 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.299288 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.299289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.299289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.299585 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.299729 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.299802 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.299927 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260929 03:30:52.302209 25821 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:52.302695 25821 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:52.305881 25821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: Generated new TSK 0
I20260929 03:30:52.306058 25821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.116.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
Entry for principal kudu/127.6.116.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab.
I20260929 03:30:52.335719 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:52.416091 25835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:52.416268 25835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:52.416290 25835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:52.417407 25835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:52.417446 25835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:52.417495 25835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:52.418919 25835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.129.keytab
--principal=kudu/127.6.116.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:52.419176 25835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:52.419404 25835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:52.424161 25840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.424163 25843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.424182 25841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:52.424353 25835 server_base.cc:1034] running on GCE node
I20260929 03:30:52.424537 25835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:52.424772 25835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:52.425926 25835 hybrid_clock.cc:648] HybridClock initialized: now 1790652652425906 us; error 31 us; skew 500 ppm
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:52.428331 25835 init.cc:377] Logged in from keytab as kudu/127.6.116.129@KRBTEST.COM (short username kudu)
I20260929 03:30:52.428735 25835 webserver.cc:460] Webserver started at http://127.6.116.129:39517/ using document root <none> and password file <none>
I20260929 03:30:52.428953 25835 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:52.429020 25835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:52.429181 25835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:52.430171 25835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "23c7130bba014ea1ae0bbe8faa15fa1f"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "49c7e16241f9c018db3b29b40230305a"
server_key_iv: "e16f1e1542cd61356673ae4a1db32f91"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.430511 25835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "23c7130bba014ea1ae0bbe8faa15fa1f"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "49c7e16241f9c018db3b29b40230305a"
server_key_iv: "e16f1e1542cd61356673ae4a1db32f91"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.431869 25835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:30:52.432576 25850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.432739 25835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:52.432827 25835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "23c7130bba014ea1ae0bbe8faa15fa1f"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "49c7e16241f9c018db3b29b40230305a"
server_key_iv: "e16f1e1542cd61356673ae4a1db32f91"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.432902 25835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:52.447649 25835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:52.449410 25835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:52.449565 25835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:52.449770 25835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:52.450058 25835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:52.450114 25835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.450152 25835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:52.450181 25835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.455675 25835 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:46347
I20260929 03:30:52.455720 25963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:46347 every 8 connection(s)
I20260929 03:30:52.455991 25835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:52.460501 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25835
I20260929 03:30:52.460618 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:52.460840 6610 external_mini_cluster.cc:1428] Setting key 63edcb486bd3ea32f111039e281a1a70
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.129@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:52.465472 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.456653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:34465 (local address 127.6.116.190:32793)
0929 03:30:52.456728 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.456730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.456906 (+ 176us) server_negotiation.cc:407] Connection header received
0929 03:30:52.457312 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.457313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.457324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.457383 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.458208 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.458754 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.459790 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.460005 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.460058 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.462564 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.462565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.462566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:52.462572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.464172 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.464557 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.464558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.464559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.464614 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.464867 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.464868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.464868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.464981 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.465057 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.465305 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.465386 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 03:30:52.465641 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.456602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:32793 (local address 127.6.116.129:34465)
0929 03:30:52.456789 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.456834 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.457235 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.457438 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.457441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.457953 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:52.458133 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.458137 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.458828 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.458828 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.459653 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.459656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.460076 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.460076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.460175 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.460688 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:52.460697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:52.462468 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:52.464261 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.464268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.464270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.464477 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.464691 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.464692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.464693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.464793 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.465051 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:52.465057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:52.465177 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.465283 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.465423 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
I20260929 03:30:52.466213 25964 heartbeater.cc:347] Connected to a master server at 127.6.116.190:32793
I20260929 03:30:52.466300 25964 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:52.466454 25964 heartbeater.cc:511] Master 127.6.116.190:32793 requested a full tablet report, sending...
I20260929 03:30:52.466929 25773 ts_manager.cc:194] Registered new tserver with Master: 23c7130bba014ea1ae0bbe8faa15fa1f (127.6.116.129:46347)
I20260929 03:30:52.467535 25773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.129@KRBTEST.COM'} at 127.6.116.129:34465
WARNING: no policy specified for kudu/127.6.116.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
Entry for principal kudu/127.6.116.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab.
I20260929 03:30:52.483919 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:52.560698 25969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:52.560873 25969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:52.560896 25969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:52.561946 25969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:52.561983 25969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:52.562008 25969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:52.563409 25969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.130.keytab
--principal=kudu/127.6.116.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:52.563576 25969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:52.563732 25969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:52.568364 25975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.568373 25974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.568373 25977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:52.568560 25969 server_base.cc:1034] running on GCE node
I20260929 03:30:52.568782 25969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:52.569070 25969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:52.570227 25969 hybrid_clock.cc:648] HybridClock initialized: now 1790652652570203 us; error 36 us; skew 500 ppm
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:52.572618 25969 init.cc:377] Logged in from keytab as kudu/127.6.116.130@KRBTEST.COM (short username kudu)
I20260929 03:30:52.573058 25969 webserver.cc:460] Webserver started at http://127.6.116.130:41891/ using document root <none> and password file <none>
I20260929 03:30:52.573276 25969 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:52.573340 25969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:52.573498 25969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:52.574455 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "b00da150824c4a9d84f8c7b60339b6ee"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "7e4657935f6b2006cabb851262332fda"
server_key_iv: "18f34031952078d4ed5caf39b597f3b8"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.574836 25969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "b00da150824c4a9d84f8c7b60339b6ee"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "7e4657935f6b2006cabb851262332fda"
server_key_iv: "18f34031952078d4ed5caf39b597f3b8"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.576134 25969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:30:52.576763 25984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.576907 25969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:52.576994 25969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "b00da150824c4a9d84f8c7b60339b6ee"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "7e4657935f6b2006cabb851262332fda"
server_key_iv: "18f34031952078d4ed5caf39b597f3b8"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.577085 25969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:52.591560 25969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:52.593230 25969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:52.593384 25969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:52.593583 25969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:52.593859 25969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:52.593923 25969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.593955 25969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:52.593981 25969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.599666 25969 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:33973
I20260929 03:30:52.599705 26097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:33973 every 8 connection(s)
I20260929 03:30:52.599965 25969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.130@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:52.607950 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 25969
I20260929 03:30:52.608059 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:52.608237 6610 external_mini_cluster.cc:1428] Setting key 546c7db975410a2ce091af38481905f0
I20260929 03:30:52.608647 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.600552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:58645 (local address 127.6.116.190:32793)
0929 03:30:52.600615 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.600617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.600802 (+ 185us) server_negotiation.cc:407] Connection header received
0929 03:30:52.601237 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.601238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.601248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.601302 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.602126 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.602623 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.603610 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.603792 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.603854 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.605707 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.605709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.605709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:52.605723 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.606988 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.607449 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.607450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.607450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.607516 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.607883 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.607884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.607885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.608059 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.608148 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.608458 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.608554 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:30:52.608800 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.600519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:32793 (local address 127.6.116.130:58645)
0929 03:30:52.600677 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.600713 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.601153 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.601368 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.601371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.601880 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:52.602045 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.602049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.602699 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.602700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.603522 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.603525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.603867 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.603868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.603955 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.604337 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:52.604344 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:52.605630 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:52.607162 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.607168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.607170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.607369 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.607577 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.607577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.607578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.607670 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.608155 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:52.608161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:52.608310 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.608426 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.608601 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260929 03:30:52.609355 26098 heartbeater.cc:347] Connected to a master server at 127.6.116.190:32793
I20260929 03:30:52.609452 26098 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:52.609625 26098 heartbeater.cc:511] Master 127.6.116.190:32793 requested a full tablet report, sending...
I20260929 03:30:52.609946 25773 ts_manager.cc:194] Registered new tserver with Master: b00da150824c4a9d84f8c7b60339b6ee (127.6.116.130:33973)
I20260929 03:30:52.610316 25773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.130@KRBTEST.COM'} at 127.6.116.130:58645
WARNING: no policy specified for kudu/127.6.116.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
Entry for principal kudu/127.6.116.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab.
I20260929 03:30:52.632038 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:52.708988 26103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:52.709206 26103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:52.709244 26103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:52.710381 26103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:52.710445 26103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:52.710489 26103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:52.711943 26103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.131.keytab
--principal=kudu/127.6.116.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:32793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:52.712198 26103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:52.712422 26103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:52.717054 26111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.717123 26109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.717141 26108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:52.717308 26103 server_base.cc:1034] running on GCE node
I20260929 03:30:52.717495 26103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:52.717721 26103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:52.718878 26103 hybrid_clock.cc:648] HybridClock initialized: now 1790652652718863 us; error 41 us; skew 500 ppm
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:52.721164 26103 init.cc:377] Logged in from keytab as kudu/127.6.116.131@KRBTEST.COM (short username kudu)
I20260929 03:30:52.721601 26103 webserver.cc:460] Webserver started at http://127.6.116.131:33701/ using document root <none> and password file <none>
I20260929 03:30:52.721761 26103 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:52.721798 26103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:52.721876 26103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:52.722724 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "a4aeb6e84232421d89802ba3979260dd"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "6413f137074772b6f4fa1c642e72b677"
server_key_iv: "5a63ffde57b974c2058c6a534cbe6507"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.723008 26103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "a4aeb6e84232421d89802ba3979260dd"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "6413f137074772b6f4fa1c642e72b677"
server_key_iv: "5a63ffde57b974c2058c6a534cbe6507"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.724220 26103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:52.724841 26118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.724982 26103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.725063 26103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "a4aeb6e84232421d89802ba3979260dd"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "6413f137074772b6f4fa1c642e72b677"
server_key_iv: "5a63ffde57b974c2058c6a534cbe6507"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.725121 26103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:52.751974 26103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:52.753877 26103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:52.753999 26103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:52.754180 26103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:52.754436 26103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:52.754465 26103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.754487 26103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:52.754498 26103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.760414 26103 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:33555
I20260929 03:30:52.760483 26231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:33555 every 8 connection(s)
I20260929 03:30:52.760689 26103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:30:52.766013 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26103
I20260929 03:30:52.766146 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:52.766350 6610 external_mini_cluster.cc:1428] Setting key 4e39db1d2d6d589cded0364e04589c5d
Sep 29 03:30:52 dist-test-slave-z2zg krb5kdc[25728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652652, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.131@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:30:52.769443 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.761307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:32989 (local address 127.6.116.190:32793)
0929 03:30:52.761371 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.761373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.761571 (+ 198us) server_negotiation.cc:407] Connection header received
0929 03:30:52.762078 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.762080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.762093 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.762168 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.763085 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.763577 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.764542 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.764738 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.764826 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.767063 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.767065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.767065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:52.767072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.768296 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.768660 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.768661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.768661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.768708 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.768907 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.768908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.768908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.769001 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.769050 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.769314 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.769384 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:30:52.769613 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.761297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:32793 (local address 127.6.116.131:32989)
0929 03:30:52.761461 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.761493 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.761972 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.762244 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.762248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.762788 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:52.762962 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.762966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.763669 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.763670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.764434 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.764436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.764848 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.764850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.764965 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.765347 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:30:52.765354 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:52.766938 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:52.768359 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.768364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.768366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.768587 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.768763 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:30:52.768764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:30:52.768764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:30:52.768855 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:30:52.769077 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:52.769083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:52.769194 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.769289 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.769414 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260929 03:30:52.769979 26232 heartbeater.cc:347] Connected to a master server at 127.6.116.190:32793
I20260929 03:30:52.770056 26232 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:52.770182 26232 heartbeater.cc:511] Master 127.6.116.190:32793 requested a full tablet report, sending...
I20260929 03:30:52.770514 25773 ts_manager.cc:194] Registered new tserver with Master: a4aeb6e84232421d89802ba3979260dd (127.6.116.131:33555)
I20260929 03:30:52.770885 25773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.116.131@KRBTEST.COM'} at 127.6.116.131:32989
I20260929 03:30:52.778021 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:52.784857 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.779398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.6.116.190:32793)
0929 03:30:52.779475 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.779477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.779573 (+ 96us) server_negotiation.cc:407] Connection header received
0929 03:30:52.779623 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.779624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.779632 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.779686 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.780209 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.780728 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.781385 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.781553 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.781603 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.782634 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.782635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.782635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:52.782641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.783948 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.784248 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.784248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.784249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.784294 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:52.784432 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:52.784432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:52.784433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:52.784545 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.784612 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.784750 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.784801 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 03:30:52.786217 25773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:52.787878 25773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:52.793617 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.789910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33973 (local address 127.0.0.1:38296)
0929 03:30:52.790062 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.790100 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.790166 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.790376 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.790379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.790380 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:52.790441 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.790442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.791505 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.791506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.792409 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.792411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.793287 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.793288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.793377 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.793389 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.793491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.793543 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260929 03:30:52.793646 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.789978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.6.116.130:33973)
0929 03:30:52.790169 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.790174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.790186 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:30:52.790254 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.790257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.790275 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.790363 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:52.790508 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.791408 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.792499 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.793217 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.793288 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.793341 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.793516 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.793558 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260929 03:30:52.794287 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.789916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:33555 (local address 127.0.0.1:56162)
0929 03:30:52.790139 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.790148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.790189 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.790557 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.790557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.790560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:52.790593 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.790594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.792036 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.792037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.793076 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.793079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.794058 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.794059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.794122 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.794126 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.794184 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.794221 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":56,"threads_started":1}
I20260929 03:30:52.794306 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.790213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.6.116.131:33555)
0929 03:30:52.790397 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.790402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.790411 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:52.790444 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.790446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.790461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.790779 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:52.790830 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.791925 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.793186 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.793932 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.794055 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.794115 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.794197 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.794231 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
I20260929 03:30:52.794482 26032 tablet_service.cc:1467] Processing CreateTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe (DEFAULT_TABLE table=test-table [id=3b37054955fb472aafef397b40c2e721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:52.794685 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.790119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:46347 (local address 127.0.0.1:35742)
0929 03:30:52.790621 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.790632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.790682 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.790909 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.790910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.790911 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:52.790949 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.790950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.792208 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.792209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.793120 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.793123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.794322 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.794323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.794507 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.794512 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.794564 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.794615 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":37,"threads_started":1}
I20260929 03:30:52.794855 26032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ff8210029429fb8be3e2b52eb9cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:52.795065 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.790451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35742 (local address 127.6.116.129:46347)
0929 03:30:52.790679 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.790683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.790694 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:30:52.790738 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.790740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.790756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.790827 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:52.791024 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.792099 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.793208 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.794171 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.794237 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.794287 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.794920 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.794969 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260929 03:30:52.794931 26166 tablet_service.cc:1467] Processing CreateTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe (DEFAULT_TABLE table=test-table [id=3b37054955fb472aafef397b40c2e721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:52.795297 26166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ff8210029429fb8be3e2b52eb9cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:52.797405 26246 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Bootstrap starting.
I20260929 03:30:52.797662 26247 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Bootstrap starting.
I20260929 03:30:52.797971 25898 tablet_service.cc:1467] Processing CreateTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe (DEFAULT_TABLE table=test-table [id=3b37054955fb472aafef397b40c2e721]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:52.798286 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ff8210029429fb8be3e2b52eb9cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:52.798295 26246 tablet_bootstrap.cc:654] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:52.798316 26247 tablet_bootstrap.cc:654] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:52.798800 26247 log.cc:824] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:52.798861 26246 log.cc:824] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:52.799616 26247 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: No bootstrap required, opened a new log
I20260929 03:30:52.799691 26247 ts_tablet_manager.cc:1397] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:52.801542 26250 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: Bootstrap starting.
I20260929 03:30:52.801679 26247 raft_consensus.cc:348] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.801858 26247 raft_consensus.cc:374] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:52.801904 26247 raft_consensus.cc:729] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4aeb6e84232421d89802ba3979260dd, State: Initialized, Role: FOLLOWER
I20260929 03:30:52.802017 26247 consensus_queue.cc:260] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.802224 26250 tablet_bootstrap.cc:654] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:52.802301 26247 ts_tablet_manager.cc:1428] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:52.802310 26232 heartbeater.cc:503] Master 127.6.116.190:32793 was elected leader, sending a full tablet report...
I20260929 03:30:52.802706 26246 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: No bootstrap required, opened a new log
I20260929 03:30:52.802775 26246 ts_tablet_manager.cc:1397] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260929 03:30:52.802789 26250 log.cc:824] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:52.803584 26250 tablet_bootstrap.cc:492] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: No bootstrap required, opened a new log
I20260929 03:30:52.803642 26250 ts_tablet_manager.cc:1397] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:52.804412 26246 raft_consensus.cc:348] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.804570 26246 raft_consensus.cc:374] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:52.804627 26246 raft_consensus.cc:729] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00da150824c4a9d84f8c7b60339b6ee, State: Initialized, Role: FOLLOWER
I20260929 03:30:52.804736 26246 consensus_queue.cc:260] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.804996 26246 ts_tablet_manager.cc:1428] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 03:30:52.805050 26098 heartbeater.cc:503] Master 127.6.116.190:32793 was elected leader, sending a full tablet report...
I20260929 03:30:52.805063 26250 raft_consensus.cc:348] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.805241 26250 raft_consensus.cc:374] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:52.805284 26250 raft_consensus.cc:729] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c7130bba014ea1ae0bbe8faa15fa1f, State: Initialized, Role: FOLLOWER
I20260929 03:30:52.805500 26250 consensus_queue.cc:260] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.805801 26250 ts_tablet_manager.cc:1428] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 03:30:52.805830 25964 heartbeater.cc:503] Master 127.6.116.190:32793 was elected leader, sending a full tablet report...
I20260929 03:30:52.806977 26253 raft_consensus.cc:482] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:52.807104 26253 raft_consensus.cc:504] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.807417 26253 leader_election.cc:288] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4aeb6e84232421d89802ba3979260dd (127.6.116.131:33555), 23c7130bba014ea1ae0bbe8faa15fa1f (127.6.116.129:46347)
I20260929 03:30:52.810151 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.807522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:33555 (local address 127.6.116.130:44685)
0929 03:30:52.807602 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.807623 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.807665 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.807802 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.807803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.807804 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:52.807860 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.807861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.808571 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.808573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.809324 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.809326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.809872 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.809949 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.809954 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.810032 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.810093 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":48}
I20260929 03:30:52.810149 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.807619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:44685 (local address 127.6.116.131:33555)
0929 03:30:52.807654 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.807657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.807664 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:30:52.807726 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.807727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.807744 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.807796 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:52.807942 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.808492 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809413 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.809779 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809871 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.809880 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.810039 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.810077 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:30:52.810247 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.807600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:55383 (local address 127.6.116.129:46347)
0929 03:30:52.807655 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.807657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.807786 (+ 129us) server_negotiation.cc:407] Connection header received
0929 03:30:52.807850 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.807851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.807859 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.807913 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:52.808085 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.808622 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809529 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.809900 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809980 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.809987 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.810160 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.810191 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:30:52.810491 26255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.807556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:46347 (local address 127.6.116.130:55383)
0929 03:30:52.807701 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.807710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:52.807800 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:52.807917 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:52.807917 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:52.807919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:52.807982 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.807984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.808703 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.808704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.809439 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:52.809441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.809991 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.809992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:52.810083 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.810086 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:52.810156 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:30:52.810191 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260929 03:30:52.810637 26186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669ff8210029429fb8be3e2b52eb9cfe" candidate_uuid: "b00da150824c4a9d84f8c7b60339b6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aeb6e84232421d89802ba3979260dd" is_pre_election: true
I20260929 03:30:52.810765 26186 raft_consensus.cc:2454] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00da150824c4a9d84f8c7b60339b6ee in term 0.
I20260929 03:30:52.810992 25988 leader_election.cc:302] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4aeb6e84232421d89802ba3979260dd, b00da150824c4a9d84f8c7b60339b6ee; no voters:
I20260929 03:30:52.811076 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669ff8210029429fb8be3e2b52eb9cfe" candidate_uuid: "b00da150824c4a9d84f8c7b60339b6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" is_pre_election: true
I20260929 03:30:52.811159 26253 raft_consensus.cc:2781] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:52.811192 25918 raft_consensus.cc:2454] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b00da150824c4a9d84f8c7b60339b6ee in term 0.
I20260929 03:30:52.811210 26253 raft_consensus.cc:482] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:52.811239 26253 raft_consensus.cc:3037] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:52.811875 26253 raft_consensus.cc:504] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.812002 26253 leader_election.cc:288] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [CANDIDATE]: Term 1 election: Requested vote from peers a4aeb6e84232421d89802ba3979260dd (127.6.116.131:33555), 23c7130bba014ea1ae0bbe8faa15fa1f (127.6.116.129:46347)
I20260929 03:30:52.812139 26186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669ff8210029429fb8be3e2b52eb9cfe" candidate_uuid: "b00da150824c4a9d84f8c7b60339b6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4aeb6e84232421d89802ba3979260dd"
I20260929 03:30:52.812137 25918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669ff8210029429fb8be3e2b52eb9cfe" candidate_uuid: "b00da150824c4a9d84f8c7b60339b6ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f"
I20260929 03:30:52.812206 25918 raft_consensus.cc:3037] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:52.812206 26186 raft_consensus.cc:3037] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:52.812978 25918 raft_consensus.cc:2454] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00da150824c4a9d84f8c7b60339b6ee in term 1.
I20260929 03:30:52.812978 26186 raft_consensus.cc:2454] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b00da150824c4a9d84f8c7b60339b6ee in term 1.
I20260929 03:30:52.813131 25988 leader_election.cc:302] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4aeb6e84232421d89802ba3979260dd, b00da150824c4a9d84f8c7b60339b6ee; no voters:
I20260929 03:30:52.813252 26253 raft_consensus.cc:2781] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:52.813377 26253 raft_consensus.cc:686] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 1 LEADER]: Becoming Leader. State: Replica: b00da150824c4a9d84f8c7b60339b6ee, State: Running, Role: LEADER
I20260929 03:30:52.813463 26253 consensus_queue.cc:237] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } }
I20260929 03:30:52.814121 25773 catalog_manager.cc:5495] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee reported cstate change: term changed from 0 to 1, leader changed from <none> to b00da150824c4a9d84f8c7b60339b6ee (127.6.116.130). New cstate: current_term: 1 leader_uuid: "b00da150824c4a9d84f8c7b60339b6ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00da150824c4a9d84f8c7b60339b6ee" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 } health_report { overall_health: UNKNOWN } } }
I20260929 03:30:52.820536 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.818485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.6.116.130:33973)
0929 03:30:52.818530 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.818531 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.818538 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:30:52.818621 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.818622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.818635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.818688 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:52.818783 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.819264 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.819962 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.820132 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.820179 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.820315 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:52.820390 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:52.820444 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.820480 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.820494 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:30:52.821987 26186 raft_consensus.cc:1261] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 1 FOLLOWER]: Refusing update from remote peer b00da150824c4a9d84f8c7b60339b6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:52.821998 25918 raft_consensus.cc:1261] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f [term 1 FOLLOWER]: Refusing update from remote peer b00da150824c4a9d84f8c7b60339b6ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:52.822270 26256 consensus_queue.cc:1035] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c7130bba014ea1ae0bbe8faa15fa1f" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:52.822417 26253 consensus_queue.cc:1035] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4aeb6e84232421d89802ba3979260dd" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:52.825387 26260 mvcc.cc:205] Tried to move back new op lower bound from 7334513265956716544 to 7334513265924448256. Current Snapshot: MvccSnapshot[applied={T|T < 7334513265956716544}]
I20260929 03:30:52.826614 25772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:52.826699 25772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:52.827370 25772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f26480ecb60844d39bf1490590ab24bf: Sending DeleteTablet for 3 replicas of tablet 669ff8210029429fb8be3e2b52eb9cfe
I20260929 03:30:52.827653 26032 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:52 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:38296
I20260929 03:30:52.827683 26166 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:52 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:56162
I20260929 03:30:52.827898 26270 tablet_replica.cc:330] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: stopping tablet replica
I20260929 03:30:52.827881 25898 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ff8210029429fb8be3e2b52eb9cfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:52 UTC) from {username='kudu', principal='kudu/127.6.116.190@KRBTEST.COM'} at 127.0.0.1:35742
I20260929 03:30:52.828049 26271 tablet_replica.cc:330] T 669ff8210029429fb8be3e2b52eb9cfe P 23c7130bba014ea1ae0bbe8faa15fa1f: stopping tablet replica
I20260929 03:30:52.828056 26270 raft_consensus.cc:2229] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:52.828060 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25835
I20260929 03:30:52.829658 26269 tablet_replica.cc:330] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: stopping tablet replica
I20260929 03:30:52.829752 26269 raft_consensus.cc:2229] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:52.829830 26269 raft_consensus.cc:2258] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:52.830268 26269 ts_tablet_manager.cc:1905] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:52.830505 26270 raft_consensus.cc:2258] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:52.830962 26270 ts_tablet_manager.cc:1905] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:52.831810 26269 ts_tablet_manager.cc:1918] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:52.831884 26269 log.cc:1196] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-1/wal/wals/669ff8210029429fb8be3e2b52eb9cfe
I20260929 03:30:52.832172 26269 ts_tablet_manager.cc:1939] T 669ff8210029429fb8be3e2b52eb9cfe P b00da150824c4a9d84f8c7b60339b6ee: Deleting consensus metadata
I20260929 03:30:52.832546 25758 catalog_manager.cc:4841] TS b00da150824c4a9d84f8c7b60339b6ee (127.6.116.130:33973): tablet 669ff8210029429fb8be3e2b52eb9cfe (table test-table [id=3b37054955fb472aafef397b40c2e721]) successfully deleted
I20260929 03:30:52.833179 26270 ts_tablet_manager.cc:1918] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:52.833250 26270 log.cc:1196] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/669ff8210029429fb8be3e2b52eb9cfe
I20260929 03:30:52.833545 26270 ts_tablet_manager.cc:1939] T 669ff8210029429fb8be3e2b52eb9cfe P a4aeb6e84232421d89802ba3979260dd: Deleting consensus metadata
I20260929 03:30:52.834148 25760 catalog_manager.cc:4841] TS a4aeb6e84232421d89802ba3979260dd (127.6.116.131:33555): tablet 669ff8210029429fb8be3e2b52eb9cfe (table test-table [id=3b37054955fb472aafef397b40c2e721]) successfully deleted
W20260929 03:30:52.834417 25758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:46347 (error 108)
I20260929 03:30:52.834884 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.834760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:46347 (local address 127.0.0.1:35748)
0929 03:30:52.834815 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:52.834856 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:46347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:30:52.834954 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25969
W20260929 03:30:52.836337 25758 catalog_manager.cc:4584] TS 23c7130bba014ea1ae0bbe8faa15fa1f (127.6.116.129:46347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 669ff8210029429fb8be3e2b52eb9cfe: Network error: Client connection negotiation failed: client connection to 127.6.116.129:46347: connect: Connection refused (error 111)
I20260929 03:30:52.838980 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26103
I20260929 03:30:52.842808 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 25741
2026-09-29T03:30:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1790 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T03:30:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:52Z Disabled control of system clock
I20260929 03:30:52.868289 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:37175
--webserver_interface=127.6.116.190
--webserver_port=0
--builtin_ntp_servers=127.6.116.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:37175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:52.945937 26276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:52.946125 26276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:52.946146 26276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:52.947172 26276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:52.947202 26276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:52.947214 26276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:52.947227 26276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:52.947237 26276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:52.948431 26276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:37175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:37175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:52.948603 26276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:52.948796 26276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:52.953218 26284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.953266 26282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:52.953245 26281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:52.953418 26276 server_base.cc:1034] running on GCE node
I20260929 03:30:52.953665 26276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:52.953917 26276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:52.955088 26276 hybrid_clock.cc:648] HybridClock initialized: now 1790652652955067 us; error 33 us; skew 500 ppm
I20260929 03:30:52.956336 26276 webserver.cc:460] Webserver started at http://127.6.116.190:40291/ using document root <none> and password file <none>
I20260929 03:30:52.956573 26276 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:52.956617 26276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:52.956751 26276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:52.957732 26276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "f9b999ed857542f098b6b87e5a81f032"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "0980ee36f6a70cb7e277ce948df4cc2b"
server_key_iv: "cef18d548aad616c0377c793d788aae2"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.958068 26276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "f9b999ed857542f098b6b87e5a81f032"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "0980ee36f6a70cb7e277ce948df4cc2b"
server_key_iv: "cef18d548aad616c0377c793d788aae2"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.959314 26276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:52.959998 26290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:52.960197 26276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:52.960261 26276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "f9b999ed857542f098b6b87e5a81f032"
format_stamp: "Formatted at 2026-09-29 03:30:52 on dist-test-slave-z2zg"
server_key: "0980ee36f6a70cb7e277ce948df4cc2b"
server_key_iv: "cef18d548aad616c0377c793d788aae2"
server_key_version: "encryptionkey@0"
I20260929 03:30:52.960341 26276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:52.981477 26276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:52.981729 26276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:52.981866 26276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:52.985481 26276 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:37175
I20260929 03:30:52.985531 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:37175 every 8 connection(s)
I20260929 03:30:52.985824 26276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:52.986701 26343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:52.988690 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: Bootstrap starting.
I20260929 03:30:52.989217 26343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:52.989529 26343 log.cc:824] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:52.990162 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: No bootstrap required, opened a new log
I20260929 03:30:52.991691 26343 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } }
I20260929 03:30:52.991856 26343 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:52.991901 26343 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b999ed857542f098b6b87e5a81f032, State: Initialized, Role: FOLLOWER
I20260929 03:30:52.991925 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26276
I20260929 03:30:52.992023 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:52.992082 26343 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } }
I20260929 03:30:52.992162 26343 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:52.992214 26343 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:52.992242 6610 external_mini_cluster.cc:1428] Setting key 23aac41cdc8d269dc85de4bea7dee601
I20260929 03:30:52.992259 26343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:52.992967 26343 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } }
I20260929 03:30:52.993072 26343 leader_election.cc:302] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b999ed857542f098b6b87e5a81f032; no voters:
I20260929 03:30:52.993214 26343 leader_election.cc:288] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:52.993281 26348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:52.993412 26348 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [term 1 LEADER]: Becoming Leader. State: Replica: f9b999ed857542f098b6b87e5a81f032, State: Running, Role: LEADER
I20260929 03:30:52.993438 26343 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:52.993492 26348 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } }
I20260929 03:30:52.993810 26348 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b999ed857542f098b6b87e5a81f032. Latest consensus state: current_term: 1 leader_uuid: "f9b999ed857542f098b6b87e5a81f032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } } }
I20260929 03:30:52.993818 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b999ed857542f098b6b87e5a81f032" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b999ed857542f098b6b87e5a81f032" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 37175 } } }
I20260929 03:30:52.993892 26348 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:52.993896 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032 [sys.catalog]: This master's current role is: LEADER
W20260929 03:30:52.995253 26364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:30:52.995314 26364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:30:52.995368 26355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:52.995517 26355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:52.996973 26355 catalog_manager.cc:1347] Generated new cluster ID: 21d03f6798a84084ba879f93ff24a8b0
I20260929 03:30:52.997022 26355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:52.998600 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:52.992733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33530 (local address 127.6.116.190:37175)
0929 03:30:52.992964 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:52.992972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:52.993005 (+ 33us) server_negotiation.cc:407] Connection header received
0929 03:30:52.993120 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:52.993127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:52.993200 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:52.993341 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:52.993720 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.995542 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.996458 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:52.997003 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:52.997132 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:52.997938 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:52.997942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:52.997942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:52.998111 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:52.998141 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:52.998150 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:52.998220 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:52.998293 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 03:30:52.998456 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260929 03:30:53.005610 26355 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:53.006256 26355 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:53.013100 26355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: Generated new TSK 0
I20260929 03:30:53.013247 26355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:53.014356 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.129:0
--local_ip_for_outbound_sockets=127.6.116.129
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--builtin_ntp_servers=127.6.116.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:53.091845 26367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:53.092043 26367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:53.092080 26367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:53.093242 26367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:53.093298 26367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:53.093338 26367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.129
I20260929 03:30:53.094805 26367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.116.129
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.129
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:53.095034 26367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:53.095333 26367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:53.100786 26375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:53.100807 26373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:53.100988 26367 server_base.cc:1034] running on GCE node
W20260929 03:30:53.100836 26372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:53.101219 26367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:53.101536 26367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:53.102690 26367 hybrid_clock.cc:648] HybridClock initialized: now 1790652653102675 us; error 43 us; skew 500 ppm
I20260929 03:30:53.104063 26367 webserver.cc:460] Webserver started at http://127.6.116.129:37269/ using document root <none> and password file <none>
I20260929 03:30:53.104339 26367 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:53.104383 26367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:53.104516 26367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:53.105525 26367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data/instance:
uuid: "4e1f57a0203946488b5c59da4d9a9c52"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "32c1256d542b50478413662e92a97017"
server_key_iv: "7874996298999031fc27f0dcfa600ab7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.105899 26367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance:
uuid: "4e1f57a0203946488b5c59da4d9a9c52"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "32c1256d542b50478413662e92a97017"
server_key_iv: "7874996298999031fc27f0dcfa600ab7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.107137 26367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:53.107880 26381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.108052 26367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:30:53.108150 26367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal
uuid: "4e1f57a0203946488b5c59da4d9a9c52"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "32c1256d542b50478413662e92a97017"
server_key_iv: "7874996298999031fc27f0dcfa600ab7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.108227 26367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:53.123970 26367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:53.124219 26367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:53.124365 26367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:53.124557 26367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:53.124841 26367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:53.124897 26367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.124938 26367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:53.124982 26367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.130437 26367 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.129:34589
I20260929 03:30:53.130471 26494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.129:34589 every 8 connection(s)
I20260929 03:30:53.130770 26367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/data/info.pb
I20260929 03:30:53.135835 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.131463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.129:56553 (local address 127.6.116.190:37175)
0929 03:30:53.131519 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.131521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.131742 (+ 221us) server_negotiation.cc:407] Connection header received
0929 03:30:53.132206 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.132207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.132220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.132273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:53.132968 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.133445 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.134427 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.134631 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.134705 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.135540 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:53.135542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:53.135542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:53.135547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:53.135561 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:53.135565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:53.135639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.135729 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.135759 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 03:30:53.136026 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.131452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:37175 (local address 127.6.116.129:56553)
0929 03:30:53.131625 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.131655 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.132109 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.132337 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.132341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.132350 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:53.132847 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.132851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.133539 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.133540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.134322 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.134324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.134715 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.134716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.134839 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.135381 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:53.135382 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:53.135406 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:53.135426 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:53.135639 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:53.135645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:53.135648 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.135730 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.135821 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260929 03:30:53.136463 26495 heartbeater.cc:347] Connected to a master server at 127.6.116.190:37175
I20260929 03:30:53.136564 26495 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:53.136742 26495 heartbeater.cc:511] Master 127.6.116.190:37175 requested a full tablet report, sending...
I20260929 03:30:53.137192 26307 ts_manager.cc:194] Registered new tserver with Master: 4e1f57a0203946488b5c59da4d9a9c52 (127.6.116.129:34589)
I20260929 03:30:53.137759 26307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.129:56553
I20260929 03:30:53.139029 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26367
I20260929 03:30:53.139180 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-0/wal/instance
I20260929 03:30:53.139371 6610 external_mini_cluster.cc:1428] Setting key 18eb0f477e017a6dae394c04b8835a3d
I20260929 03:30:53.140196 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.130:0
--local_ip_for_outbound_sockets=127.6.116.130
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--builtin_ntp_servers=127.6.116.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 03:30:53.215507 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:53.215713 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:53.215746 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:53.216778 26498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:53.216835 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:53.216876 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.130
I20260929 03:30:53.218061 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.116.130
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.130
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:53.218276 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:53.218528 26498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:53.223089 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:53.223145 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:53.223258 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:53.223371 26498 server_base.cc:1034] running on GCE node
I20260929 03:30:53.223611 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:53.223837 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:53.224992 26498 hybrid_clock.cc:648] HybridClock initialized: now 1790652653224974 us; error 29 us; skew 500 ppm
I20260929 03:30:53.226125 26498 webserver.cc:460] Webserver started at http://127.6.116.130:37491/ using document root <none> and password file <none>
I20260929 03:30:53.226356 26498 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:53.226421 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:53.226550 26498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:53.227594 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data/instance:
uuid: "5871380172574e99acc43ec29c12207f"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "b8480c891fcdf821d5c4a083478ef130"
server_key_iv: "5db92a1a303924b1b92669d20e6a57a7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.227952 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance:
uuid: "5871380172574e99acc43ec29c12207f"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "b8480c891fcdf821d5c4a083478ef130"
server_key_iv: "5db92a1a303924b1b92669d20e6a57a7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.229197 26498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:53.229833 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.229996 26498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:53.230086 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal
uuid: "5871380172574e99acc43ec29c12207f"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "b8480c891fcdf821d5c4a083478ef130"
server_key_iv: "5db92a1a303924b1b92669d20e6a57a7"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.230167 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:53.273726 26498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:53.273980 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:53.274119 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:53.274320 26498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:53.274659 26498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:53.274701 26498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.274735 26498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:53.274765 26498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.280058 26498 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.130:33363
I20260929 03:30:53.280099 26625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.130:33363 every 8 connection(s)
I20260929 03:30:53.280380 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/data/info.pb
I20260929 03:30:53.284761 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.281008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.130:43823 (local address 127.6.116.190:37175)
0929 03:30:53.281076 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.281078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.281339 (+ 261us) server_negotiation.cc:407] Connection header received
0929 03:30:53.281760 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.281761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.281771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.281820 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:53.282394 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.282847 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.283706 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.283903 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.283947 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.284548 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:53.284549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:53.284549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:53.284552 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:53.284563 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:53.284567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:53.284604 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.284690 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.284709 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:30:53.284857 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26498
I20260929 03:30:53.284951 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-1/wal/instance
I20260929 03:30:53.284989 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.281001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:37175 (local address 127.6.116.130:43823)
0929 03:30:53.281205 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.281235 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.281680 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.281867 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.281870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.281877 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:53.282329 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.282333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.282914 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.282915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.283640 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.283642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.283965 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.283965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.284057 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.284438 (+ 381us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:53.284438 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:53.284461 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:53.284479 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:53.284627 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:53.284630 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:53.284632 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.284678 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.284785 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260929 03:30:53.285179 6610 external_mini_cluster.cc:1428] Setting key 926226a335e7d20bffee8aa96da4db1a
I20260929 03:30:53.285395 26626 heartbeater.cc:347] Connected to a master server at 127.6.116.190:37175
I20260929 03:30:53.285487 26626 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:53.285655 26626 heartbeater.cc:511] Master 127.6.116.190:37175 requested a full tablet report, sending...
I20260929 03:30:53.286010 26307 ts_manager.cc:194] Registered new tserver with Master: 5871380172574e99acc43ec29c12207f (127.6.116.130:33363)
I20260929 03:30:53.286140 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.116.131:0
--local_ip_for_outbound_sockets=127.6.116.131
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--builtin_ntp_servers=127.6.116.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 03:30:53.286471 26307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.130:43823
W20260929 03:30:53.362902 26629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:53.363103 26629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:53.363170 26629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:53.364342 26629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:53.364375 26629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:53.364398 26629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.116.131
I20260929 03:30:53.365607 26629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.116.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.116.131
--webserver_port=0
--tserver_master_addrs=127.6.116.190:37175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.116.131
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:53.365863 26629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:53.366139 26629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:53.371150 26635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:53.371169 26637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:53.371510 26629 server_base.cc:1034] running on GCE node
W20260929 03:30:53.371948 26634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:53.372171 26629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:53.372414 26629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:53.373581 26629 hybrid_clock.cc:648] HybridClock initialized: now 1790652653373566 us; error 29 us; skew 500 ppm
I20260929 03:30:53.374782 26629 webserver.cc:460] Webserver started at http://127.6.116.131:44137/ using document root <none> and password file <none>
I20260929 03:30:53.375022 26629 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:53.375116 26629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:53.375252 26629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:53.376186 26629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data/instance:
uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "cb164967f64ea82d6c6595c816e7fe95"
server_key_iv: "79c726ad7e8fa58faee60edb568ff023"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.376520 26629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance:
uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "cb164967f64ea82d6c6595c816e7fe95"
server_key_iv: "79c726ad7e8fa58faee60edb568ff023"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.377671 26629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:53.378317 26643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.378484 26629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:53.378573 26629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal
uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8"
format_stamp: "Formatted at 2026-09-29 03:30:53 on dist-test-slave-z2zg"
server_key: "cb164967f64ea82d6c6595c816e7fe95"
server_key_iv: "79c726ad7e8fa58faee60edb568ff023"
server_key_version: "encryptionkey@0"
I20260929 03:30:53.378659 26629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:53.405965 26629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:53.406234 26629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:53.406379 26629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:53.406585 26629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 03:30:53.406888 26629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 03:30:53.406939 26629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.406971 26629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 03:30:53.406996 26629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:53.412770 26629 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.131:33933
I20260929 03:30:53.412815 26756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.131:33933 every 8 connection(s)
I20260929 03:30:53.413094 26629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/data/info.pb
I20260929 03:30:53.417805 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.413780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:40607 (local address 127.6.116.190:37175)
0929 03:30:53.413852 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.413854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.414178 (+ 324us) server_negotiation.cc:407] Connection header received
0929 03:30:53.414626 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.414627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.414636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.414685 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:53.415333 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.415795 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.416675 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.416845 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.416892 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.417590 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:53.417591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:53.417591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:53.417595 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:53.417606 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:53.417610 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:53.417652 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.417731 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.417752 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 03:30:53.417973 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.413748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:37175 (local address 127.6.116.131:40607)
0929 03:30:53.414010 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.414055 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.414541 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.414750 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.414753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.414761 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:30:53.415245 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.415249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.415872 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.415872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.416603 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.416605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.416923 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.416924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.417026 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.417463 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 03:30:53.417463 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:30:53.417488 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 03:30:53.417507 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:30:53.417663 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:30:53.417667 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:30:53.417669 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.417717 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.417786 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260929 03:30:53.418402 26757 heartbeater.cc:347] Connected to a master server at 127.6.116.190:37175
I20260929 03:30:53.418505 26757 heartbeater.cc:464] Registering TS with master...
I20260929 03:30:53.418694 26757 heartbeater.cc:511] Master 127.6.116.190:37175 requested a full tablet report, sending...
I20260929 03:30:53.419109 26307 ts_manager.cc:194] Registered new tserver with Master: f36cf0e9b4404cdb9c77427bbb2d9aa8 (127.6.116.131:33933)
I20260929 03:30:53.419435 26307 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.116.131:40607
I20260929 03:30:53.421052 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26629
I20260929 03:30:53.421164 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal/instance
I20260929 03:30:53.421361 6610 external_mini_cluster.cc:1428] Setting key e13c634ddc648207464fbfe23ccdd4bf
I20260929 03:30:53.422161 6610 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 03:30:53.425875 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.423449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.6.116.190:37175)
0929 03:30:53.423498 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.423500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.423568 (+ 68us) server_negotiation.cc:407] Connection header received
0929 03:30:53.423621 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.423621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.423630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.423673 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:53.423850 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.424256 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.424895 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.425055 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.425100 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.425676 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:53.425677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:53.425677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:53.425680 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:53.425689 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:53.425692 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:53.425740 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.425812 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.425829 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 03:30:53.427241 26307 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 03:30:53.428665 26307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 03:30:53.434687 26765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33363 (local address 127.0.0.1:34776)
0929 03:30:53.430827 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.430863 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.430922 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.431495 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.431496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.431498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:53.431561 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.431562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432416 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.432417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.433565 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.433568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434424 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.434497 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434510 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.434578 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.434622 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":56,"threads_started":1}
I20260929 03:30:53.434723 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.6.116.130:33363)
0929 03:30:53.431214 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.431217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.431224 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:30:53.431297 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.431299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.431398 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.431636 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:53.431686 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432331 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.433649 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434319 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434418 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434501 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.434592 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.434634 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":70,"threads_started":1}
I20260929 03:30:53.435078 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.6.116.131:33933)
0929 03:30:53.431144 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.431148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.431330 (+ 182us) server_negotiation.cc:407] Connection header received
0929 03:30:53.431376 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.431377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.431390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.431464 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:53.431633 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432694 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.433983 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434633 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434700 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434772 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.434945 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.434984 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
I20260929 03:30:53.435240 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.6.116.129:34589)
0929 03:30:53.431135 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.431140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.431151 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:30:53.431192 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.431196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.431212 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.431305 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:53.431570 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432750 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.433808 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434321 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434421 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434503 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.435099 (+ 596us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.435154 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":145,"threads_started":1}
I20260929 03:30:53.435367 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34589 (local address 127.0.0.1:57056)
0929 03:30:53.430994 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.431004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.431085 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.431396 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.431398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.431401 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:53.431454 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.431455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432828 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.432829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.433694 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.433697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434848 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.434900 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434904 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.434947 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.434976 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":71,"threads_started":1}
I20260929 03:30:53.435510 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.430716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.131:33933 (local address 127.0.0.1:41978)
0929 03:30:53.431175 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.431185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.431268 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.431530 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.431531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.431532 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:53.431561 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.431562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.432778 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.432779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.433905 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.433907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.434715 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.434716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.434863 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.434867 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.434920 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.435022 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":70,"threads_started":1}
I20260929 03:30:53.435581 26560 tablet_service.cc:1467] Processing CreateTablet for tablet 3fed831e3ddb416087c16481004cae23 (DEFAULT_TABLE table=test-table [id=9d4d388cbbaf4565b946fb3a95f4e5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:53.435964 26560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fed831e3ddb416087c16481004cae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:53.436084 26429 tablet_service.cc:1467] Processing CreateTablet for tablet 3fed831e3ddb416087c16481004cae23 (DEFAULT_TABLE table=test-table [id=9d4d388cbbaf4565b946fb3a95f4e5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:53.436228 26691 tablet_service.cc:1467] Processing CreateTablet for tablet 3fed831e3ddb416087c16481004cae23 (DEFAULT_TABLE table=test-table [id=9d4d388cbbaf4565b946fb3a95f4e5d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 03:30:53.436403 26429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fed831e3ddb416087c16481004cae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:53.436506 26691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fed831e3ddb416087c16481004cae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:53.438548 26771 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: Bootstrap starting.
I20260929 03:30:53.439103 26772 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Bootstrap starting.
I20260929 03:30:53.439256 26771 tablet_bootstrap.cc:654] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:53.439620 26773 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Bootstrap starting.
I20260929 03:30:53.439755 26772 tablet_bootstrap.cc:654] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:53.439757 26771 log.cc:824] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:53.440050 26772 log.cc:824] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:53.440215 26773 tablet_bootstrap.cc:654] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:53.440510 26773 log.cc:824] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:53.440618 26771 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: No bootstrap required, opened a new log
I20260929 03:30:53.440701 26771 ts_tablet_manager.cc:1397] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:53.440798 26772 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: No bootstrap required, opened a new log
I20260929 03:30:53.440860 26772 ts_tablet_manager.cc:1397] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:53.441414 26773 tablet_bootstrap.cc:492] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: No bootstrap required, opened a new log
I20260929 03:30:53.441493 26773 ts_tablet_manager.cc:1397] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 03:30:53.442778 26771 raft_consensus.cc:348] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.442915 26771 raft_consensus.cc:374] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:53.442955 26771 raft_consensus.cc:729] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1f57a0203946488b5c59da4d9a9c52, State: Initialized, Role: FOLLOWER
I20260929 03:30:53.443080 26771 consensus_queue.cc:260] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.443336 26771 ts_tablet_manager.cc:1428] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:53.443367 26772 raft_consensus.cc:348] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.443478 26495 heartbeater.cc:503] Master 127.6.116.190:37175 was elected leader, sending a full tablet report...
I20260929 03:30:53.443502 26772 raft_consensus.cc:374] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:53.443550 26772 raft_consensus.cc:729] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5871380172574e99acc43ec29c12207f, State: Initialized, Role: FOLLOWER
I20260929 03:30:53.443645 26772 consensus_queue.cc:260] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.443888 26772 ts_tablet_manager.cc:1428] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:53.443918 26626 heartbeater.cc:503] Master 127.6.116.190:37175 was elected leader, sending a full tablet report...
I20260929 03:30:53.443986 26773 raft_consensus.cc:348] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.444155 26773 raft_consensus.cc:374] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:53.444216 26773 raft_consensus.cc:729] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36cf0e9b4404cdb9c77427bbb2d9aa8, State: Initialized, Role: FOLLOWER
I20260929 03:30:53.444339 26773 consensus_queue.cc:260] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.444591 26773 ts_tablet_manager.cc:1428] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 03:30:53.444703 26757 heartbeater.cc:503] Master 127.6.116.190:37175 was elected leader, sending a full tablet report...
I20260929 03:30:53.526083 26779 raft_consensus.cc:482] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 03:30:53.526183 26779 raft_consensus.cc:504] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.526470 26779 leader_election.cc:288] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5871380172574e99acc43ec29c12207f (127.6.116.130:33363), 4e1f57a0203946488b5c59da4d9a9c52 (127.6.116.129:34589)
I20260929 03:30:53.530243 26770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.526703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:46405 (local address 127.6.116.129:34589)
0929 03:30:53.526747 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.526749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.526903 (+ 154us) server_negotiation.cc:407] Connection header received
0929 03:30:53.526935 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.526936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.526945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.526995 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:53.527172 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.527863 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529345 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.529888 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529974 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.529981 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.530138 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.530178 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 03:30:53.530243 26769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.526704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.116.131:54285 (local address 127.6.116.130:33363)
0929 03:30:53.526757 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.526759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.526766 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:30:53.526801 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.526802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.526810 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.526860 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 03:30:53.527078 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.527835 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529346 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.529888 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529974 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.529981 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.530150 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.530184 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:30:53.530246 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.526636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33363 (local address 127.6.116.131:54285)
0929 03:30:53.526691 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.526703 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.526742 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.526894 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.526895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.526897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:53.526970 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.526972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.527928 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.527929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.529256 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.529258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.529984 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.530060 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.530063 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.530134 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.530166 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":44}
I20260929 03:30:53.530258 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.526647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34589 (local address 127.6.116.131:46405)
0929 03:30:53.526804 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.526812 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 03:30:53.526893 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:30:53.527019 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:30:53.527020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:30:53.527022 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 03:30:53.527100 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.527101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.527931 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.527949 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.529256 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:30:53.529258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.529986 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.529986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:30:53.530063 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.530065 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 03:30:53.530118 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:30:53.530160 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":102,"thread_start_us":83,"threads_started":1}
I20260929 03:30:53.530632 26580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fed831e3ddb416087c16481004cae23" candidate_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5871380172574e99acc43ec29c12207f" is_pre_election: true
I20260929 03:30:53.530637 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fed831e3ddb416087c16481004cae23" candidate_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1f57a0203946488b5c59da4d9a9c52" is_pre_election: true
I20260929 03:30:53.530779 26580 raft_consensus.cc:2454] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36cf0e9b4404cdb9c77427bbb2d9aa8 in term 0.
I20260929 03:30:53.530795 26449 raft_consensus.cc:2454] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36cf0e9b4404cdb9c77427bbb2d9aa8 in term 0.
W20260929 03:30:53.530838 26627 tablet.cc:2369] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:30:53.530968 26647 leader_election.cc:302] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5871380172574e99acc43ec29c12207f, f36cf0e9b4404cdb9c77427bbb2d9aa8; no voters:
I20260929 03:30:53.531095 26779 raft_consensus.cc:2781] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 03:30:53.531167 26779 raft_consensus.cc:482] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 03:30:53.531190 26779 raft_consensus.cc:3037] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:53.531862 26779 raft_consensus.cc:504] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.532001 26779 leader_election.cc:288] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [CANDIDATE]: Term 1 election: Requested vote from peers 5871380172574e99acc43ec29c12207f (127.6.116.130:33363), 4e1f57a0203946488b5c59da4d9a9c52 (127.6.116.129:34589)
I20260929 03:30:53.532155 26449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fed831e3ddb416087c16481004cae23" candidate_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1f57a0203946488b5c59da4d9a9c52"
I20260929 03:30:53.532226 26449 raft_consensus.cc:3037] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:53.532234 26580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fed831e3ddb416087c16481004cae23" candidate_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5871380172574e99acc43ec29c12207f"
I20260929 03:30:53.532316 26580 raft_consensus.cc:3037] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:53.532965 26449 raft_consensus.cc:2454] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36cf0e9b4404cdb9c77427bbb2d9aa8 in term 1.
I20260929 03:30:53.533116 26580 raft_consensus.cc:2454] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36cf0e9b4404cdb9c77427bbb2d9aa8 in term 1.
I20260929 03:30:53.533129 26646 leader_election.cc:302] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1f57a0203946488b5c59da4d9a9c52, f36cf0e9b4404cdb9c77427bbb2d9aa8; no voters:
I20260929 03:30:53.533216 26779 raft_consensus.cc:2781] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:53.533397 26779 raft_consensus.cc:686] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 1 LEADER]: Becoming Leader. State: Replica: f36cf0e9b4404cdb9c77427bbb2d9aa8, State: Running, Role: LEADER
I20260929 03:30:53.533499 26779 consensus_queue.cc:237] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } }
I20260929 03:30:53.534235 26306 catalog_manager.cc:5495] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f36cf0e9b4404cdb9c77427bbb2d9aa8 (127.6.116.131). New cstate: current_term: 1 leader_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f36cf0e9b4404cdb9c77427bbb2d9aa8" member_type: VOTER last_known_addr { host: "127.6.116.131" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260929 03:30:53.551380 26768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.548931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.6.116.131:33933)
0929 03:30:53.548977 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:53.548979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:53.548988 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:30:53.549065 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:53.549065 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:53.549081 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:53.549165 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:30:53.549270 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.549712 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.550567 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:53.550759 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:53.550841 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:53.551001 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:30:53.551142 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:30:53.551236 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:53.551274 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 03:30:53.551291 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 03:30:53.552914 26580 raft_consensus.cc:1261] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 1 FOLLOWER]: Refusing update from remote peer f36cf0e9b4404cdb9c77427bbb2d9aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:53.552906 26449 raft_consensus.cc:1261] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 1 FOLLOWER]: Refusing update from remote peer f36cf0e9b4404cdb9c77427bbb2d9aa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:30:53.553233 26781 consensus_queue.cc:1035] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5871380172574e99acc43ec29c12207f" member_type: VOTER last_known_addr { host: "127.6.116.130" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:53.553357 26779 consensus_queue.cc:1035] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e1f57a0203946488b5c59da4d9a9c52" member_type: VOTER last_known_addr { host: "127.6.116.129" port: 34589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:30:53.557818 26306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:53.557904 26306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 03:30:53.558646 26306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9b999ed857542f098b6b87e5a81f032: Sending DeleteTablet for 3 replicas of tablet 3fed831e3ddb416087c16481004cae23
I20260929 03:30:53.558935 26691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fed831e3ddb416087c16481004cae23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:53 UTC) from {username='slave'} at 127.0.0.1:41978
I20260929 03:30:53.558931 26429 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fed831e3ddb416087c16481004cae23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:53 UTC) from {username='slave'} at 127.0.0.1:57056
I20260929 03:30:53.558961 26560 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fed831e3ddb416087c16481004cae23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 03:30:53 UTC) from {username='slave'} at 127.0.0.1:34776
I20260929 03:30:53.559175 26794 tablet_replica.cc:330] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: stopping tablet replica
I20260929 03:30:53.559185 26793 tablet_replica.cc:330] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52: stopping tablet replica
I20260929 03:30:53.559253 26794 raft_consensus.cc:2229] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 03:30:53.559269 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26367
I20260929 03:30:53.559273 26793 raft_consensus.cc:2229] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:53.559324 26794 raft_consensus.cc:2258] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:53.559360 26793 raft_consensus.cc:2258] T 3fed831e3ddb416087c16481004cae23 P 4e1f57a0203946488b5c59da4d9a9c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:53.559844 26794 ts_tablet_manager.cc:1905] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 03:30:53.561336 26794 ts_tablet_manager.cc:1918] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 03:30:53.561394 26794 log.cc:1196] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Deleting WAL directory at /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790652420563207-6610-0/minicluster-data/ts-2/wal/wals/3fed831e3ddb416087c16481004cae23
I20260929 03:30:53.561640 26794 ts_tablet_manager.cc:1939] T 3fed831e3ddb416087c16481004cae23 P f36cf0e9b4404cdb9c77427bbb2d9aa8: Deleting consensus metadata
I20260929 03:30:53.562076 26292 catalog_manager.cc:4841] TS f36cf0e9b4404cdb9c77427bbb2d9aa8 (127.6.116.131:33933): tablet 3fed831e3ddb416087c16481004cae23 (table test-table [id=9d4d388cbbaf4565b946fb3a95f4e5d9]) successfully deleted
W20260929 03:30:53.565019 26293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.116.129:34589 (error 108)
I20260929 03:30:53.565207 26795 tablet_replica.cc:330] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f: stopping tablet replica
I20260929 03:30:53.565304 26795 raft_consensus.cc:2229] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 03:30:53.565402 26795 raft_consensus.cc:2258] T 3fed831e3ddb416087c16481004cae23 P 5871380172574e99acc43ec29c12207f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 03:30:53.565461 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.565338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.129:34589 (local address 127.0.0.1:57072)
0929 03:30:53.565379 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.565424 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.129:34589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 03:30:53.565575 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26498
W20260929 03:30:53.566180 26293 catalog_manager.cc:4584] TS 4e1f57a0203946488b5c59da4d9a9c52 (127.6.116.129:34589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fed831e3ddb416087c16481004cae23: Network error: Client connection negotiation failed: client connection to 127.6.116.129:34589: connect: Connection refused (error 111)
I20260929 03:30:53.570539 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:53.570277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.130:33363 (local address 127.0.0.1:34786)
0929 03:30:53.570493 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 03:30:53.570517 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.130:33363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260929 03:30:53.570565 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26629
I20260929 03:30:53.574898 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26276
2026-09-29T03:30:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (732 ms)
[----------] 6 tests from EncryptionPolicyTest (7856 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:53 dist-test-slave-z2zg krb5kdc[26799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:53 dist-test-slave-z2zg krb5kdc[26799](info): set up 2 sockets
Sep 29 03:30:53 dist-test-slave-z2zg krb5kdc[26799](info): commencing operation
krb5kdc: starting...
W20260929 03:30:54.588752 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:30:54.697201 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:54.773835 26812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:54.774045 26812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:54.774081 26812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:54.775177 26812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:54.775243 26812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:54.775267 26812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:54.775292 26812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:54.775322 26812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:54.776690 26812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42175
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:54.776930 26812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:54.777133 26812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:54.781749 26817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:54.781770 26820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:54.781774 26812 server_base.cc:1034] running on GCE node
W20260929 03:30:54.781759 26818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:54.782212 26812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:54.782483 26812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:54.783627 26812 hybrid_clock.cc:648] HybridClock initialized: now 1790652654783613 us; error 27 us; skew 500 ppm
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:54.785945 26812 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:30:54.786298 26812 webserver.cc:460] Webserver started at http://127.0.0.1:40853/ using document root <none> and password file <none>
I20260929 03:30:54.786480 26812 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:54.786517 26812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:54.786598 26812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:54.787547 26812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "a78e5e3ea4dd4d1695a145439f2e70f2"
format_stamp: "Formatted at 2026-09-29 03:30:54 on dist-test-slave-z2zg"
server_key: "f97e4983e2cd08f83812cecfa96f435f"
server_key_iv: "662e23350ed0bcdb58362af05e7bb280"
server_key_version: "encryptionkey@0"
I20260929 03:30:54.787832 26812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "a78e5e3ea4dd4d1695a145439f2e70f2"
format_stamp: "Formatted at 2026-09-29 03:30:54 on dist-test-slave-z2zg"
server_key: "f97e4983e2cd08f83812cecfa96f435f"
server_key_iv: "662e23350ed0bcdb58362af05e7bb280"
server_key_version: "encryptionkey@0"
I20260929 03:30:54.788970 26812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:54.789568 26827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:54.789745 26812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:54.789846 26812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "a78e5e3ea4dd4d1695a145439f2e70f2"
format_stamp: "Formatted at 2026-09-29 03:30:54 on dist-test-slave-z2zg"
server_key: "f97e4983e2cd08f83812cecfa96f435f"
server_key_iv: "662e23350ed0bcdb58362af05e7bb280"
server_key_version: "encryptionkey@0"
I20260929 03:30:54.789925 26812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:54.801993 26812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:54.804118 26812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:54.804263 26812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:54.807824 26812 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42175
I20260929 03:30:54.807852 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42175 every 8 connection(s)
I20260929 03:30:54.808178 26812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:54.809186 26880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:54.810906 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26812
I20260929 03:30:54.811018 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:54.811240 6610 external_mini_cluster.cc:1428] Setting key d35463a9c8e722d21238e4e583456975
I20260929 03:30:54.811491 26880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2: Bootstrap starting.
I20260929 03:30:54.812155 26880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:54.812515 26880 log.cc:824] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:54.813154 26880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2: No bootstrap required, opened a new log
I20260929 03:30:54.814790 26880 raft_consensus.cc:348] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260929 03:30:54.814905 26880 raft_consensus.cc:374] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:54.814926 26880 raft_consensus.cc:729] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78e5e3ea4dd4d1695a145439f2e70f2, State: Initialized, Role: FOLLOWER
I20260929 03:30:54.815088 26880 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260929 03:30:54.815166 26880 raft_consensus.cc:388] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:54.815187 26880 raft_consensus.cc:482] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:54.815249 26880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:54.815732 26880 raft_consensus.cc:504] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260929 03:30:54.815834 26880 leader_election.cc:302] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78e5e3ea4dd4d1695a145439f2e70f2; no voters:
I20260929 03:30:54.815991 26880 leader_election.cc:288] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:54.816156 26885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:54.816238 26880 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:54.816352 26885 raft_consensus.cc:686] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [term 1 LEADER]: Becoming Leader. State: Replica: a78e5e3ea4dd4d1695a145439f2e70f2, State: Running, Role: LEADER
I20260929 03:30:54.816447 26885 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } }
I20260929 03:30:54.816816 26886 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260929 03:30:54.816833 26887 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78e5e3ea4dd4d1695a145439f2e70f2. Latest consensus state: current_term: 1 leader_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78e5e3ea4dd4d1695a145439f2e70f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42175 } } }
I20260929 03:30:54.816916 26886 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:54.816924 26887 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:54.817090 26894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:54.817564 26894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:30:54.819568 26894 catalog_manager.cc:1347] Generated new cluster ID: c411c950cae6465fa2a1715c8d4b41b9
I20260929 03:30:54.819628 26894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:54.822008 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:54.811713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47497 (local address 127.0.0.1:42175)
0929 03:30:54.811856 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:54.811863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:54.811887 (+ 24us) server_negotiation.cc:407] Connection header received
0929 03:30:54.812453 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:54.812459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:54.812538 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:54.812705 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:54.813313 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:54.814183 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:54.814953 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:54.815503 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:54.815630 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:54.818455 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:54.818459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:54.818460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:54.818470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:54.820342 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:54.820762 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:54.820765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:54.820765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:54.820870 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:54.821095 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:54.821096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:54.821096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:54.821423 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:54.821584 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:54.821706 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 03:30:54.821807 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260929 03:30:54.822971 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:30:54.830983 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:54.824302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41819 (local address 127.0.0.1:42175)
0929 03:30:54.824333 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:54.824335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:54.824393 (+ 58us) server_negotiation.cc:407] Connection header received
0929 03:30:54.824444 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:54.824445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:54.824453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:54.824517 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:54.825030 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:54.825791 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:54.826741 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:54.827116 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:54.827187 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:54.828375 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:54.828376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:54.828376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:54.828383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:54.829834 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:54.830222 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:54.830223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:54.830223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:54.830278 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:54.830484 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:54.830484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:54.830485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:54.830598 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:54.830711 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:54.830824 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:30:54.830882 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:30:54.834643 26894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:54.835358 26894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:54.848007 26894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a78e5e3ea4dd4d1695a145439f2e70f2: Generated new TSK 0
I20260929 03:30:54.848158 26894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:54.853099 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26812
W20260929 03:30:54.857594 26797 connection.cc:535] client connection to 127.0.0.1:42175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42175): Connection reset by peer (error 104)
2026-09-29T03:30:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26911](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26911](info): set up 2 sockets
Sep 29 03:30:54 dist-test-slave-z2zg krb5kdc[26911](info): commencing operation
krb5kdc: starting...
W20260929 03:30:55.856662 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:55 dist-test-slave-z2zg krb5kdc[26911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:30:55.964120 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:56.041481 26924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:56.041695 26924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:56.041735 26924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:56.042848 26924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:56.042915 26924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:56.042943 26924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:56.042976 26924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:56.042999 26924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:56.044453 26924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:56.044705 26924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:56.044926 26924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:56.049633 26932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:56.049576 26930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:56.049606 26929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:56.049885 26924 server_base.cc:1034] running on GCE node
I20260929 03:30:56.050118 26924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:56.050426 26924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:56.051578 26924 hybrid_clock.cc:648] HybridClock initialized: now 1790652656051559 us; error 32 us; skew 500 ppm
Sep 29 03:30:56 dist-test-slave-z2zg krb5kdc[26911](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:56.060382 26924 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:30:56.060791 26924 webserver.cc:460] Webserver started at http://127.0.0.1:42843/ using document root <none> and password file <none>
I20260929 03:30:56.061060 26924 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:56.061134 26924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:56.061287 26924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:56.062314 26924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "cca7efaac17d4e04b451c1aeea937bb6"
format_stamp: "Formatted at 2026-09-29 03:30:56 on dist-test-slave-z2zg"
server_key: "ef6ffc52474cf8f7b5173c484748d448"
server_key_iv: "6694516173d0d1d2ba130ce5a4945911"
server_key_version: "encryptionkey@0"
I20260929 03:30:56.062683 26924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "cca7efaac17d4e04b451c1aeea937bb6"
format_stamp: "Formatted at 2026-09-29 03:30:56 on dist-test-slave-z2zg"
server_key: "ef6ffc52474cf8f7b5173c484748d448"
server_key_iv: "6694516173d0d1d2ba130ce5a4945911"
server_key_version: "encryptionkey@0"
I20260929 03:30:56.064222 26924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:30:56.064962 26939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:56.065171 26924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:30:56.065266 26924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "cca7efaac17d4e04b451c1aeea937bb6"
format_stamp: "Formatted at 2026-09-29 03:30:56 on dist-test-slave-z2zg"
server_key: "ef6ffc52474cf8f7b5173c484748d448"
server_key_iv: "6694516173d0d1d2ba130ce5a4945911"
server_key_version: "encryptionkey@0"
I20260929 03:30:56.065341 26924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:56.082490 26924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:56.084263 26924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:56.084417 26924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:56.088197 26924 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41857
I20260929 03:30:56.088222 26991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41857 every 8 connection(s)
I20260929 03:30:56.088544 26924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:56.089430 26992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:56.091449 26992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6: Bootstrap starting.
I20260929 03:30:56.091915 26992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:56.092147 26992 log.cc:824] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:56.092749 26992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6: No bootstrap required, opened a new log
I20260929 03:30:56.094143 26992 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260929 03:30:56.094259 26992 raft_consensus.cc:374] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:56.094280 26992 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca7efaac17d4e04b451c1aeea937bb6, State: Initialized, Role: FOLLOWER
I20260929 03:30:56.094399 26992 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260929 03:30:56.094480 26992 raft_consensus.cc:388] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:56.094502 26992 raft_consensus.cc:482] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:56.094528 26992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:56.094956 26992 raft_consensus.cc:504] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260929 03:30:56.095031 26992 leader_election.cc:302] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cca7efaac17d4e04b451c1aeea937bb6; no voters:
I20260929 03:30:56.095208 26992 leader_election.cc:288] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:56.095278 26995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:56.095438 26995 raft_consensus.cc:686] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [term 1 LEADER]: Becoming Leader. State: Replica: cca7efaac17d4e04b451c1aeea937bb6, State: Running, Role: LEADER
I20260929 03:30:56.095459 26992 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:56.095522 26995 consensus_queue.cc:237] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } }
I20260929 03:30:56.096042 26996 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cca7efaac17d4e04b451c1aeea937bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } } }
I20260929 03:30:56.096141 26996 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:56.096470 27004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:56.096568 26997 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cca7efaac17d4e04b451c1aeea937bb6. Latest consensus state: current_term: 1 leader_uuid: "cca7efaac17d4e04b451c1aeea937bb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cca7efaac17d4e04b451c1aeea937bb6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41857 } } }
I20260929 03:30:56.096657 26997 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:56.096910 27004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:56.098387 27004 catalog_manager.cc:1347] Generated new cluster ID: ff30015ba48249db8eec5281cd8a69a1
I20260929 03:30:56.098443 27004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:56.098453 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 26924
I20260929 03:30:56.098552 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:56.098729 6610 external_mini_cluster.cc:1428] Setting key c545d6786d66d2dd9f3d16626d62fe62
I20260929 03:30:56.108482 27004 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:56.108732 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:56.099198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34971 (local address 127.0.0.1:41857)
0929 03:30:56.099360 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:56.099368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:56.103375 (+ 4007us) server_negotiation.cc:407] Connection header received
0929 03:30:56.103516 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:56.103525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:56.103586 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:56.103725 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:56.103884 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:56.105358 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:56.106430 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:56.107023 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:56.107185 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:56.107880 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:56.107884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:56.107885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:56.108081 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:56.108124 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:56.108138 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:56.108224 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:56.108454 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 03:30:56.108483 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260929 03:30:56.109179 27004 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:56.109508 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:30:56.112330 27004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cca7efaac17d4e04b451c1aeea937bb6: Generated new TSK 0
I20260929 03:30:56.112475 27004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:56.113853 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:56.110759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58129 (local address 127.0.0.1:41857)
0929 03:30:56.110794 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:56.110796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:56.110891 (+ 95us) server_negotiation.cc:407] Connection header received
0929 03:30:56.110956 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:56.110957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:56.110966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:56.111008 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:56.111146 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:56.111649 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:56.112767 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:56.112979 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:56.113056 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:56.113633 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:56.113634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:56.113634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:56.113638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:56.113649 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:56.113653 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:56.113702 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:56.113758 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 03:30:56.113768 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:30:56.115577 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 26924
W20260929 03:30:56.119535 26909 connection.cc:535] client connection to 127.0.0.1:41857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41857): Connection reset by peer (error 104)
2026-09-29T03:30:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1262 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:56 dist-test-slave-z2zg krb5kdc[27023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:56 dist-test-slave-z2zg krb5kdc[27023](info): set up 2 sockets
Sep 29 03:30:56 dist-test-slave-z2zg krb5kdc[27023](info): commencing operation
krb5kdc: starting...
W20260929 03:30:57.112732 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:57 dist-test-slave-z2zg krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:30:57.223608 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:57.299469 27036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:57.299641 27036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:57.299664 27036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:57.300678 27036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:57.300712 27036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:57.300726 27036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:57.300738 27036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:57.300750 27036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:57.302073 27036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:57.302234 27036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:57.302385 27036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:57.306887 27044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:57.306893 27041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:57.306947 27042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:57.306941 27036 server_base.cc:1034] running on GCE node
I20260929 03:30:57.307417 27036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:57.307677 27036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:57.308830 27036 hybrid_clock.cc:648] HybridClock initialized: now 1790652657308817 us; error 25 us; skew 500 ppm
Sep 29 03:30:57 dist-test-slave-z2zg krb5kdc[27023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:57.311132 27036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:30:57.311532 27036 webserver.cc:460] Webserver started at http://127.0.0.1:36605/ using document root <none> and password file <none>
I20260929 03:30:57.311731 27036 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:57.311770 27036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:57.311852 27036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:57.312755 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "a90030d760074c6d9442fc135e9ce965"
format_stamp: "Formatted at 2026-09-29 03:30:57 on dist-test-slave-z2zg"
server_key: "632a4ae30a9322d628f26111dcb51f42"
server_key_iv: "fda5882ad332e23b0153e0be1f5959b0"
server_key_version: "encryptionkey@0"
I20260929 03:30:57.313026 27036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "a90030d760074c6d9442fc135e9ce965"
format_stamp: "Formatted at 2026-09-29 03:30:57 on dist-test-slave-z2zg"
server_key: "632a4ae30a9322d628f26111dcb51f42"
server_key_iv: "fda5882ad332e23b0153e0be1f5959b0"
server_key_version: "encryptionkey@0"
I20260929 03:30:57.314170 27036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:30:57.314801 27051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:57.314982 27036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:57.315089 27036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "a90030d760074c6d9442fc135e9ce965"
format_stamp: "Formatted at 2026-09-29 03:30:57 on dist-test-slave-z2zg"
server_key: "632a4ae30a9322d628f26111dcb51f42"
server_key_iv: "fda5882ad332e23b0153e0be1f5959b0"
server_key_version: "encryptionkey@0"
I20260929 03:30:57.315155 27036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:57.318137 27036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:57.320170 27036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:57.320308 27036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:57.323968 27036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38307
I20260929 03:30:57.323995 27103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38307 every 8 connection(s)
I20260929 03:30:57.324276 27036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:57.325129 27104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:57.327204 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965: Bootstrap starting.
I20260929 03:30:57.327442 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27036
I20260929 03:30:57.327546 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:57.327713 6610 external_mini_cluster.cc:1428] Setting key 490060c920b908fc02d84b3bf69f3568
I20260929 03:30:57.327865 27104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:57.328213 27104 log.cc:824] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:57.328866 27104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965: No bootstrap required, opened a new log
I20260929 03:30:57.329998 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:57.328238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41283 (local address 127.0.0.1:38307)
0929 03:30:57.328424 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:57.328432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:57.328460 (+ 28us) server_negotiation.cc:407] Connection header received
0929 03:30:57.328867 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:57.328873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:57.328893 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:57.329043 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:57.329586 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:57.329589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:57.329590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:57.329593 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:57.329624 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:57.329656 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:57.329742 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:57.329814 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 03:30:57.329850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260929 03:30:57.330827 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:30:57.331370 27104 raft_consensus.cc:348] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 03:30:57.331522 27104 raft_consensus.cc:374] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:57.331588 27104 raft_consensus.cc:729] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90030d760074c6d9442fc135e9ce965, State: Initialized, Role: FOLLOWER
I20260929 03:30:57.331694 27104 consensus_queue.cc:260] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 03:30:57.331805 27104 raft_consensus.cc:388] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:57.331839 27104 raft_consensus.cc:482] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:57.331938 27104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:57.332530 27104 raft_consensus.cc:504] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 03:30:57.332645 27104 leader_election.cc:302] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a90030d760074c6d9442fc135e9ce965; no voters:
I20260929 03:30:57.332805 27104 leader_election.cc:288] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:57.332968 27114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:57.333055 27104 sys_catalog.cc:564] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:57.333192 27114 raft_consensus.cc:686] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [term 1 LEADER]: Becoming Leader. State: Replica: a90030d760074c6d9442fc135e9ce965, State: Running, Role: LEADER
I20260929 03:30:57.333346 27107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:57.332235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50679 (local address 127.0.0.1:38307)
0929 03:30:57.332269 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:57.332270 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:57.332419 (+ 149us) server_negotiation.cc:407] Connection header received
0929 03:30:57.332473 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:57.332474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:57.332477 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:57.332527 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:57.333116 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:57.333117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:57.333117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:57.333122 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:57.333133 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:57.333137 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:57.333166 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:57.333279 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:30:57.333290 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:30:57.333285 27114 consensus_queue.cc:237] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } }
I20260929 03:30:57.333835 27115 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a90030d760074c6d9442fc135e9ce965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } } }
I20260929 03:30:57.333889 27115 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:57.333813 27114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a90030d760074c6d9442fc135e9ce965. Latest consensus state: current_term: 1 leader_uuid: "a90030d760074c6d9442fc135e9ce965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a90030d760074c6d9442fc135e9ce965" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38307 } } }
I20260929 03:30:57.334092 27114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:57.334081 27123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:57.334555 27123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:57.335968 27123 catalog_manager.cc:1347] Generated new cluster ID: 66400d29d6bc4212a9f5b3195d8dcd64
I20260929 03:30:57.336018 27123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:57.357120 27123 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:57.357280 27123 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:57.361511 27123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a90030d760074c6d9442fc135e9ce965: Generated new TSK 0
I20260929 03:30:57.361699 27123 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:57.363816 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27036
2026-09-29T03:30:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1247 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:57 dist-test-slave-z2zg krb5kdc[27135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:57 dist-test-slave-z2zg krb5kdc[27135](info): set up 2 sockets
Sep 29 03:30:57 dist-test-slave-z2zg krb5kdc[27135](info): commencing operation
krb5kdc: starting...
W20260929 03:30:58.372819 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:30:58.479765 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:58.555397 27148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:58.555567 27148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:58.555588 27148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:58.556623 27148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:58.556654 27148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:58.556668 27148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:58.556680 27148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:58.556691 27148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:58.558075 27148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:58.558246 27148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:58.558408 27148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:58.562760 27153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:58.562824 27154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:58.562767 27156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:58.562974 27148 server_base.cc:1034] running on GCE node
I20260929 03:30:58.563206 27148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:58.563464 27148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:58.564599 27148 hybrid_clock.cc:648] HybridClock initialized: now 1790652658564589 us; error 26 us; skew 500 ppm
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:58.566783 27148 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:30:58.567247 27148 webserver.cc:460] Webserver started at http://127.0.0.1:41767/ using document root <none> and password file <none>
I20260929 03:30:58.567461 27148 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:58.567521 27148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:58.567646 27148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:58.568565 27148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "dfa65bfd478b4cdb8d445d8adea0a954"
format_stamp: "Formatted at 2026-09-29 03:30:58 on dist-test-slave-z2zg"
server_key: "18f689f6f3f1b4f5ae0dca4f3290fb6b"
server_key_iv: "e9086be84312b1a5ea6e342cd45e33c1"
server_key_version: "encryptionkey@0"
I20260929 03:30:58.568888 27148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "dfa65bfd478b4cdb8d445d8adea0a954"
format_stamp: "Formatted at 2026-09-29 03:30:58 on dist-test-slave-z2zg"
server_key: "18f689f6f3f1b4f5ae0dca4f3290fb6b"
server_key_iv: "e9086be84312b1a5ea6e342cd45e33c1"
server_key_version: "encryptionkey@0"
I20260929 03:30:58.570076 27148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:30:58.570705 27163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:58.570860 27148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:58.570968 27148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "dfa65bfd478b4cdb8d445d8adea0a954"
format_stamp: "Formatted at 2026-09-29 03:30:58 on dist-test-slave-z2zg"
server_key: "18f689f6f3f1b4f5ae0dca4f3290fb6b"
server_key_iv: "e9086be84312b1a5ea6e342cd45e33c1"
server_key_version: "encryptionkey@0"
I20260929 03:30:58.571074 27148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:58.597695 27148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:58.599283 27148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:58.599421 27148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:58.603149 27148 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43465
I20260929 03:30:58.603197 27215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43465 every 8 connection(s)
I20260929 03:30:58.603482 27148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:58.604346 27216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:58.604422 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27148
I20260929 03:30:58.604550 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:58.604732 6610 external_mini_cluster.cc:1428] Setting key 32dca3dcd9db9edf8427e06518bad141
I20260929 03:30:58.607487 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954: Bootstrap starting.
I20260929 03:30:58.608085 27216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:58.608372 27216 log.cc:824] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:58.608973 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954: No bootstrap required, opened a new log
I20260929 03:30:58.610571 27216 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260929 03:30:58.610742 27216 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:58.610774 27216 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa65bfd478b4cdb8d445d8adea0a954, State: Initialized, Role: FOLLOWER
I20260929 03:30:58.610899 27216 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260929 03:30:58.610972 27216 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:58.611001 27216 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:58.611065 27216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:30:58.611552 27216 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260929 03:30:58.611635 27216 leader_election.cc:302] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfa65bfd478b4cdb8d445d8adea0a954; no voters:
I20260929 03:30:58.611845 27216 leader_election.cc:288] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:58.611872 27221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:58.611982 27221 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [term 1 LEADER]: Becoming Leader. State: Replica: dfa65bfd478b4cdb8d445d8adea0a954, State: Running, Role: LEADER
I20260929 03:30:58.612057 27221 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } }
I20260929 03:30:58.612087 27216 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:58.612581 27222 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfa65bfd478b4cdb8d445d8adea0a954. Latest consensus state: current_term: 1 leader_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } } }
I20260929 03:30:58.612680 27222 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:58.612908 27230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:58.612480 27223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfa65bfd478b4cdb8d445d8adea0a954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43465 } } }
I20260929 03:30:58.613395 27230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:58.613409 27223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954 [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:58.614897 27230 catalog_manager.cc:1347] Generated new cluster ID: 1a845536ded4470b903946e76835f594
I20260929 03:30:58.614941 27230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:58.615185 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:58.605187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45839 (local address 127.0.0.1:43465)
0929 03:30:58.605360 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:58.605370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:58.605400 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:30:58.605877 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:58.605885 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:58.605943 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:58.606077 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:58.606626 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:58.607948 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:58.608752 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:58.609186 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:58.609304 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:58.611542 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:58.611546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:58.611546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:58.611556 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:58.613603 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:58.613993 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:58.613996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:58.613996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:58.614086 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:58.614281 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:58.614282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:58.614282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:58.614566 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:58.614711 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:58.614806 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 03:30:58.614954 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260929 03:30:58.619683 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:30:58.626427 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:58.620904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37907 (local address 127.0.0.1:43465)
0929 03:30:58.620935 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:58.620938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:58.620964 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:30:58.621028 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:58.621028 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:58.621036 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:58.621085 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:58.621559 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:58.622006 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:58.622649 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:58.622870 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:58.622934 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:58.624036 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:58.624037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:58.624037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:30:58.624043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:58.625333 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:58.625666 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:58.625667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:58.625667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:58.625734 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:30:58.625935 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:30:58.625936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:30:58.625936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:30:58.626037 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:58.626123 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:58.626267 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 03:30:58.626332 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:30:58.638871 27230 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:58.639384 27230 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:58.641953 27230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfa65bfd478b4cdb8d445d8adea0a954: Generated new TSK 0
I20260929 03:30:58.642120 27230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:58.647861 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27148
2026-09-29T03:30:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27247](info): set up 2 sockets
Sep 29 03:30:58 dist-test-slave-z2zg krb5kdc[27247](info): commencing operation
krb5kdc: starting...
W20260929 03:30:59.640740 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:30:59 dist-test-slave-z2zg krb5kdc[27247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:30:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:30:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:30:59.747973 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39499
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:30:59.824117 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:30:59.824325 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:30:59.824347 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:30:59.825459 27260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:30:59.825500 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:30:59.825513 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:30:59.825526 27260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:30:59.825577 27260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:30:59.826961 27260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:30:59.827212 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:30:59.827417 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:30:59.831879 27266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:59.831920 27268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:30:59.831902 27265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:30:59.832159 27260 server_base.cc:1034] running on GCE node
I20260929 03:30:59.832453 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:30:59.832752 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:30:59.833904 27260 hybrid_clock.cc:648] HybridClock initialized: now 1790652659833876 us; error 43 us; skew 500 ppm
Sep 29 03:30:59 dist-test-slave-z2zg krb5kdc[27247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652659, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:30:59.836295 27260 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:30:59.836715 27260 webserver.cc:460] Webserver started at http://127.0.0.1:44959/ using document root <none> and password file <none>
I20260929 03:30:59.836931 27260 fs_manager.cc:359] Metadata directory not provided
I20260929 03:30:59.837003 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:30:59.837154 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:30:59.838124 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "2b82507431b94315ae734b1697313f9e"
format_stamp: "Formatted at 2026-09-29 03:30:59 on dist-test-slave-z2zg"
server_key: "1f803854b54727c0ab9ac40bea5c3b2f"
server_key_iv: "748fc5dd14d72ed4f0196f92023d89f3"
server_key_version: "encryptionkey@0"
I20260929 03:30:59.838485 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "2b82507431b94315ae734b1697313f9e"
format_stamp: "Formatted at 2026-09-29 03:30:59 on dist-test-slave-z2zg"
server_key: "1f803854b54727c0ab9ac40bea5c3b2f"
server_key_iv: "748fc5dd14d72ed4f0196f92023d89f3"
server_key_version: "encryptionkey@0"
I20260929 03:30:59.839857 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:30:59.840574 27275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:30:59.840759 27260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:30:59.840873 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "2b82507431b94315ae734b1697313f9e"
format_stamp: "Formatted at 2026-09-29 03:30:59 on dist-test-slave-z2zg"
server_key: "1f803854b54727c0ab9ac40bea5c3b2f"
server_key_iv: "748fc5dd14d72ed4f0196f92023d89f3"
server_key_version: "encryptionkey@0"
I20260929 03:30:59.840967 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:30:59.861385 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:30:59.863322 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:30:59.863462 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:30:59.867484 27260 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39499
I20260929 03:30:59.867511 27327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39499 every 8 connection(s)
I20260929 03:30:59.867878 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:30:59.868764 27328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:30:59.870818 27328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e: Bootstrap starting.
I20260929 03:30:59.871419 27328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e: Neither blocks nor log segments found. Creating new log.
I20260929 03:30:59.871686 27328 log.cc:824] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:30:59.872004 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27260
I20260929 03:30:59.872129 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:30:59.872339 6610 external_mini_cluster.cc:1428] Setting key 35aa127e9f6d0dea81b0ee21c0761105
I20260929 03:30:59.872431 27328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e: No bootstrap required, opened a new log
I20260929 03:30:59.874105 27328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260929 03:30:59.874250 27328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:30:59.874270 27328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b82507431b94315ae734b1697313f9e, State: Initialized, Role: FOLLOWER
I20260929 03:30:59.874449 27328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260929 03:30:59.874529 27328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:30:59.874598 27328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:30:59.874651 27328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:30:59.875200 27328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260929 03:30:59.875279 27328 leader_election.cc:302] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b82507431b94315ae734b1697313f9e; no voters:
I20260929 03:30:59.875478 27328 leader_election.cc:288] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:30:59.875502 27333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:30:59.875626 27333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [term 1 LEADER]: Becoming Leader. State: Replica: 2b82507431b94315ae734b1697313f9e, State: Running, Role: LEADER
I20260929 03:30:59.875726 27328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:30:59.875756 27333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } }
I20260929 03:30:59.876153 27333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b82507431b94315ae734b1697313f9e. Latest consensus state: current_term: 1 leader_uuid: "2b82507431b94315ae734b1697313f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } } }
I20260929 03:30:59.876165 27335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b82507431b94315ae734b1697313f9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b82507431b94315ae734b1697313f9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39499 } } }
I20260929 03:30:59.876289 27333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:59.876330 27335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e [sys.catalog]: This master's current role is: LEADER
I20260929 03:30:59.876750 27340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:30:59.877451 27340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:30:59.878926 27340 catalog_manager.cc:1347] Generated new cluster ID: 2fed7f62108b44319b72bb708b48fc6c
I20260929 03:30:59.878984 27340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:30:59.879570 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:59.872834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40657 (local address 127.0.0.1:39499)
0929 03:30:59.873012 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:59.873021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:59.873059 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:30:59.874094 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:59.874102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:59.874150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:59.874304 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:59.874558 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:59.876069 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:59.877252 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:59.877922 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:59.878056 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:59.878733 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:59.878737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:59.878737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:59.878967 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:59.879008 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:59.879022 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:59.879117 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:59.879306 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 03:30:59.879374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260929 03:30:59.880522 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:30:59.885294 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:30:59.882051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58743 (local address 127.0.0.1:39499)
0929 03:30:59.882085 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:30:59.882087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:30:59.882095 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:30:59.882148 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:30:59.882149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:30:59.882158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:30:59.882214 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:30:59.882336 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:59.883013 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:59.884101 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:30:59.884378 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:30:59.884438 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:30:59.885057 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:30:59.885058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:30:59.885059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:30:59.885064 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:30:59.885077 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:30:59.885082 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:30:59.885141 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:30:59.885206 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:30:59.885227 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:30:59.890762 27340 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:30:59.891433 27340 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:30:59.904510 27340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b82507431b94315ae734b1697313f9e: Generated new TSK 0
I20260929 03:30:59.904691 27340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:30:59.913270 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27260
2026-09-29T03:30:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1265 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:30:59 dist-test-slave-z2zg krb5kdc[27359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:30:59 dist-test-slave-z2zg krb5kdc[27359](info): set up 2 sockets
Sep 29 03:30:59 dist-test-slave-z2zg krb5kdc[27359](info): commencing operation
krb5kdc: starting...
W20260929 03:31:00.916944 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:00 dist-test-slave-z2zg krb5kdc[27359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:01.024037 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:01.100282 27372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:01.100452 27372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:01.100473 27372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:01.101519 27372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:01.101548 27372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:01.101562 27372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:01.101573 27372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:01.101585 27372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:01.102924 27372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:01.103133 27372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:01.103282 27372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:01.107859 27380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:01.107937 27372 server_base.cc:1034] running on GCE node
W20260929 03:31:01.107913 27378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:01.107834 27377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:01.108273 27372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:01.108490 27372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:01.109627 27372 hybrid_clock.cc:648] HybridClock initialized: now 1790652661109614 us; error 29 us; skew 500 ppm
Sep 29 03:31:01 dist-test-slave-z2zg krb5kdc[27359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652661, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:01.116776 27372 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:01.117218 27372 webserver.cc:460] Webserver started at http://127.0.0.1:36737/ using document root <none> and password file <none>
I20260929 03:31:01.117440 27372 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:01.117517 27372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:01.117658 27372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:01.118588 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "2ffb772f3a4c454e83516fab964c0136"
format_stamp: "Formatted at 2026-09-29 03:31:01 on dist-test-slave-z2zg"
server_key: "b79ed6413be907ecb4f7584219665b26"
server_key_iv: "7f92706e7dd74c86b9fc02960ad4011b"
server_key_version: "encryptionkey@0"
I20260929 03:31:01.118922 27372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "2ffb772f3a4c454e83516fab964c0136"
format_stamp: "Formatted at 2026-09-29 03:31:01 on dist-test-slave-z2zg"
server_key: "b79ed6413be907ecb4f7584219665b26"
server_key_iv: "7f92706e7dd74c86b9fc02960ad4011b"
server_key_version: "encryptionkey@0"
I20260929 03:31:01.120198 27372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:01.120909 27387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:01.121074 27372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:01.121170 27372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "2ffb772f3a4c454e83516fab964c0136"
format_stamp: "Formatted at 2026-09-29 03:31:01 on dist-test-slave-z2zg"
server_key: "b79ed6413be907ecb4f7584219665b26"
server_key_iv: "7f92706e7dd74c86b9fc02960ad4011b"
server_key_version: "encryptionkey@0"
I20260929 03:31:01.121248 27372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:01.124500 27372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:01.126145 27372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:01.126302 27372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:01.130091 27372 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33613
I20260929 03:31:01.130141 27439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33613 every 8 connection(s)
I20260929 03:31:01.130409 27372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:01.131311 27440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:01.133268 27440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136: Bootstrap starting.
I20260929 03:31:01.133787 27440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:01.134065 27440 log.cc:824] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:01.134698 27440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136: No bootstrap required, opened a new log
I20260929 03:31:01.136238 27440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 03:31:01.136354 27440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:01.136375 27440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ffb772f3a4c454e83516fab964c0136, State: Initialized, Role: FOLLOWER
I20260929 03:31:01.136547 27440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 03:31:01.136615 27440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:01.136636 27440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:01.136698 27440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:01.137176 27440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 03:31:01.137275 27440 leader_election.cc:302] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ffb772f3a4c454e83516fab964c0136; no voters:
I20260929 03:31:01.137441 27440 leader_election.cc:288] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:01.137480 27443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:01.137660 27443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [term 1 LEADER]: Becoming Leader. State: Replica: 2ffb772f3a4c454e83516fab964c0136, State: Running, Role: LEADER
I20260929 03:31:01.137670 27440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:01.137758 27443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } }
I20260929 03:31:01.138332 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27372
I20260929 03:31:01.138317 27444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ffb772f3a4c454e83516fab964c0136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } } }
I20260929 03:31:01.138401 27444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:01.138446 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:01.138321 27445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ffb772f3a4c454e83516fab964c0136. Latest consensus state: current_term: 1 leader_uuid: "2ffb772f3a4c454e83516fab964c0136" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ffb772f3a4c454e83516fab964c0136" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33613 } } }
I20260929 03:31:01.138551 27445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:01.138588 27452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:01.138628 6610 external_mini_cluster.cc:1428] Setting key 9db4fc6b11c32dc69edd7268334c710c
I20260929 03:31:01.139045 27452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:01.140769 27452 catalog_manager.cc:1347] Generated new cluster ID: bfa849eb2c9d416b8b6791e8e834062f
I20260929 03:31:01.140815 27452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:01.141136 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:01.139225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37861 (local address 127.0.0.1:33613)
0929 03:31:01.139376 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:01.139385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:01.139429 (+ 44us) server_negotiation.cc:407] Connection header received
0929 03:31:01.139778 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:01.139786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:01.139825 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:01.139963 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:01.140649 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:01.140652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:01.140653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:01.140657 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:01.140698 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:01.140711 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:01.140774 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:01.140932 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 03:31:01.140967 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260929 03:31:01.141970 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:01.144244 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:01.143293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46553 (local address 127.0.0.1:33613)
0929 03:31:01.143321 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:01.143323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:01.143435 (+ 112us) server_negotiation.cc:407] Connection header received
0929 03:31:01.143477 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:01.143478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:01.143481 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:01.143564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:01.144046 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:01.144047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:01.144048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:01.144051 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:01.144064 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:01.144068 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:01.144135 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:01.144180 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 03:31:01.144192 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:31:01.159585 27452 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:01.159727 27452 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:01.166965 27452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ffb772f3a4c454e83516fab964c0136: Generated new TSK 0
I20260929 03:31:01.167145 27452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:01.169792 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27372
2026-09-29T03:31:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:01 dist-test-slave-z2zg krb5kdc[27471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:01 dist-test-slave-z2zg krb5kdc[27471](info): set up 2 sockets
Sep 29 03:31:01 dist-test-slave-z2zg krb5kdc[27471](info): commencing operation
krb5kdc: starting...
W20260929 03:31:02.189127 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:02.295612 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:02.375504 27484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:02.375702 27484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:02.375725 27484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:02.376838 27484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:02.376879 27484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:02.376894 27484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:02.376935 27484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:02.376962 27484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:02.378362 27484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:02.378615 27484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:02.378829 27484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:02.383495 27489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:02.383496 27490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:02.383668 27484 server_base.cc:1034] running on GCE node
W20260929 03:31:02.383610 27492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:02.383980 27484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:02.384282 27484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:02.385423 27484 hybrid_clock.cc:648] HybridClock initialized: now 1790652662385406 us; error 38 us; skew 500 ppm
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:02.387795 27484 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:02.388206 27484 webserver.cc:460] Webserver started at http://127.0.0.1:38325/ using document root <none> and password file <none>
I20260929 03:31:02.388419 27484 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:02.388463 27484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:02.388623 27484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:02.389606 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "2d49fc2c47624497becb59479bd30e33"
format_stamp: "Formatted at 2026-09-29 03:31:02 on dist-test-slave-z2zg"
server_key: "5333124703cf163a875f047ebd8f9196"
server_key_iv: "669659480e3c0fb21157865925e1990a"
server_key_version: "encryptionkey@0"
I20260929 03:31:02.389945 27484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "2d49fc2c47624497becb59479bd30e33"
format_stamp: "Formatted at 2026-09-29 03:31:02 on dist-test-slave-z2zg"
server_key: "5333124703cf163a875f047ebd8f9196"
server_key_iv: "669659480e3c0fb21157865925e1990a"
server_key_version: "encryptionkey@0"
I20260929 03:31:02.391248 27484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:31:02.391913 27499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:02.392120 27484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:31:02.392207 27484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "2d49fc2c47624497becb59479bd30e33"
format_stamp: "Formatted at 2026-09-29 03:31:02 on dist-test-slave-z2zg"
server_key: "5333124703cf163a875f047ebd8f9196"
server_key_iv: "669659480e3c0fb21157865925e1990a"
server_key_version: "encryptionkey@0"
I20260929 03:31:02.392275 27484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:02.408026 27484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:02.409793 27484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:02.409950 27484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:02.413832 27484 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44451
I20260929 03:31:02.413863 27551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44451 every 8 connection(s)
I20260929 03:31:02.414196 27484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:02.415179 27552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:02.417352 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33: Bootstrap starting.
I20260929 03:31:02.417953 27552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:02.418232 27552 log.cc:824] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:02.418887 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33: No bootstrap required, opened a new log
I20260929 03:31:02.419307 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27484
I20260929 03:31:02.419430 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:02.419589 6610 external_mini_cluster.cc:1428] Setting key 7919386d29e53c10ad752e5497a5bbbc
I20260929 03:31:02.420863 27552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260929 03:31:02.421025 27552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:02.421061 27552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d49fc2c47624497becb59479bd30e33, State: Initialized, Role: FOLLOWER
I20260929 03:31:02.421221 27552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260929 03:31:02.421312 27552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:02.421346 27552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:02.421399 27552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:02.422349 27552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260929 03:31:02.422434 27552 leader_election.cc:302] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d49fc2c47624497becb59479bd30e33; no voters:
I20260929 03:31:02.422612 27552 leader_election.cc:288] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:02.422652 27557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:02.422773 27557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [term 1 LEADER]: Becoming Leader. State: Replica: 2d49fc2c47624497becb59479bd30e33, State: Running, Role: LEADER
I20260929 03:31:02.422880 27552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:02.422883 27557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } }
I20260929 03:31:02.423276 27559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d49fc2c47624497becb59479bd30e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260929 03:31:02.423360 27559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:02.423568 27563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:02.424072 27563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:02.424302 27558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d49fc2c47624497becb59479bd30e33. Latest consensus state: current_term: 1 leader_uuid: "2d49fc2c47624497becb59479bd30e33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d49fc2c47624497becb59479bd30e33" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44451 } } }
I20260929 03:31:02.424561 27558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:02.426388 27563 catalog_manager.cc:1347] Generated new cluster ID: 6cb3a9aee14443b6bb5a5c5b3209b933
I20260929 03:31:02.426455 27563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:31:02.431269 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:02.420105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:59365 (local address 127.0.0.1:44451)
0929 03:31:02.420265 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:02.420273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:02.420305 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:31:02.420846 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:02.420851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:02.420897 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:02.421002 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:02.421796 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:02.422891 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:02.424038 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:02.424736 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:02.424838 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:02.427559 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:02.427563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:02.427563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:02.427574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:02.429552 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:02.429995 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:02.429998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:02.429998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:02.430120 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:02.430361 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:02.430362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:02.430362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:02.430637 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:02.430783 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:02.430970 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 03:31:02.431084 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260929 03:31:02.432265 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:02.436985 27563 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:02.437692 27563 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:02.440869 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:02.433779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:34673 (local address 127.0.0.1:44451)
0929 03:31:02.433815 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:02.433817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:02.433881 (+ 64us) server_negotiation.cc:407] Connection header received
0929 03:31:02.433952 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:02.433953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:02.433960 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:02.434002 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:02.434601 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:02.435330 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:02.436331 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:02.436623 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:02.436717 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:02.438068 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:02.438069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:02.438069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:02.438075 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:02.439563 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:02.439939 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:02.439939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:02.439940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:02.439984 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:02.440174 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:02.440174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:02.440175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:02.440277 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:02.440368 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:02.440730 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 03:31:02.440787 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":18}
I20260929 03:31:02.450378 27563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d49fc2c47624497becb59479bd30e33: Generated new TSK 0
I20260929 03:31:02.450574 27563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:02.453084 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27484
W20260929 03:31:02.457212 27469 connection.cc:535] client connection to 127.0.0.1:44451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44451): Connection reset by peer (error 104)
2026-09-29T03:31:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27583](info): set up 2 sockets
Sep 29 03:31:02 dist-test-slave-z2zg krb5kdc[27583](info): commencing operation
krb5kdc: starting...
W20260929 03:31:03.468979 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:03 dist-test-slave-z2zg krb5kdc[27583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:03.577896 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:03.654803 27596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:03.655187 27596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:03.655216 27596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:03.656312 27596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:03.656347 27596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:03.656360 27596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:03.656373 27596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:03.656385 27596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:03.657728 27596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:03.657893 27596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:03.658041 27596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:03.662482 27601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:03.662459 27602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:03.662472 27604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:03.662765 27596 server_base.cc:1034] running on GCE node
I20260929 03:31:03.663028 27596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:03.663335 27596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:03.664481 27596 hybrid_clock.cc:648] HybridClock initialized: now 1790652663664458 us; error 44 us; skew 500 ppm
Sep 29 03:31:03 dist-test-slave-z2zg krb5kdc[27583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:03.666823 27596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:03.667303 27596 webserver.cc:460] Webserver started at http://127.0.0.1:32897/ using document root <none> and password file <none>
I20260929 03:31:03.667524 27596 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:03.667567 27596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:03.667709 27596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:03.668717 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5"
format_stamp: "Formatted at 2026-09-29 03:31:03 on dist-test-slave-z2zg"
server_key: "abc45c5afd8cf9f7ecbe2f0671d8babb"
server_key_iv: "ee4b2beffb690678b96592adce6f7ece"
server_key_version: "encryptionkey@0"
I20260929 03:31:03.669062 27596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5"
format_stamp: "Formatted at 2026-09-29 03:31:03 on dist-test-slave-z2zg"
server_key: "abc45c5afd8cf9f7ecbe2f0671d8babb"
server_key_iv: "ee4b2beffb690678b96592adce6f7ece"
server_key_version: "encryptionkey@0"
I20260929 03:31:03.670388 27596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:31:03.671146 27611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:03.671336 27596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:03.671417 27596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5"
format_stamp: "Formatted at 2026-09-29 03:31:03 on dist-test-slave-z2zg"
server_key: "abc45c5afd8cf9f7ecbe2f0671d8babb"
server_key_iv: "ee4b2beffb690678b96592adce6f7ece"
server_key_version: "encryptionkey@0"
I20260929 03:31:03.671506 27596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:03.706719 27596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:03.708982 27596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:03.709091 27596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:03.712751 27596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33593
I20260929 03:31:03.712802 27663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33593 every 8 connection(s)
I20260929 03:31:03.713140 27596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:03.714073 27664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:03.716359 27664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5: Bootstrap starting.
I20260929 03:31:03.716950 27664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:03.717229 27664 log.cc:824] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:03.717902 27664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5: No bootstrap required, opened a new log
I20260929 03:31:03.719434 27664 raft_consensus.cc:348] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } }
I20260929 03:31:03.719597 27664 raft_consensus.cc:374] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:03.719666 27664 raft_consensus.cc:729] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64d9aa9407e4bec8d3ece62ff3a3cf5, State: Initialized, Role: FOLLOWER
I20260929 03:31:03.719830 27664 consensus_queue.cc:260] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } }
I20260929 03:31:03.719911 27664 raft_consensus.cc:388] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:03.719954 27664 raft_consensus.cc:482] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:03.720000 27664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:03.720486 27664 raft_consensus.cc:504] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } }
I20260929 03:31:03.720604 27664 leader_election.cc:302] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b64d9aa9407e4bec8d3ece62ff3a3cf5; no voters:
I20260929 03:31:03.720781 27664 leader_election.cc:288] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:03.720827 27667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:03.721022 27664 sys_catalog.cc:564] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:03.721035 27667 raft_consensus.cc:686] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [term 1 LEADER]: Becoming Leader. State: Replica: b64d9aa9407e4bec8d3ece62ff3a3cf5, State: Running, Role: LEADER
I20260929 03:31:03.721135 27667 consensus_queue.cc:237] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } }
I20260929 03:31:03.721695 27668 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } } }
I20260929 03:31:03.721719 27669 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64d9aa9407e4bec8d3ece62ff3a3cf5. Latest consensus state: current_term: 1 leader_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64d9aa9407e4bec8d3ece62ff3a3cf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33593 } } }
I20260929 03:31:03.721774 27668 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:03.721807 27669 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:03.722033 27676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:03.722199 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27596
I20260929 03:31:03.722316 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:03.722536 6610 external_mini_cluster.cc:1428] Setting key 81ee7670d7a6d3ddc694052c5bf29091
I20260929 03:31:03.722678 27676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:03.724416 27676 catalog_manager.cc:1347] Generated new cluster ID: e336cccbbd8d4a569a8a5d0dfc05c390
I20260929 03:31:03.724469 27676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:03.728132 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:03.723146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:37823 (local address 127.0.0.1:33593)
0929 03:31:03.723785 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:03.723792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:03.723813 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:31:03.723940 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:03.723948 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:03.724011 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:03.724165 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:03.724408 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:03.725357 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:03.726205 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:03.726619 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:03.726758 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:03.727498 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:03.727502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:03.727502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:03.727672 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:03.727708 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:03.727718 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:03.727814 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:03.727895 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:31:03.727922 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":82,"threads_started":1}
I20260929 03:31:03.729349 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:03.733963 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:03.730675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:48659 (local address 127.0.0.1:33593)
0929 03:31:03.730712 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:03.730714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:03.731206 (+ 492us) server_negotiation.cc:407] Connection header received
0929 03:31:03.731277 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:03.731277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:03.731288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:03.731343 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:03.731451 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:03.732064 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:03.732762 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:03.732990 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:03.733074 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:03.733722 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:03.733724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:03.733725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:03.733729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:03.733743 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:03.733746 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:03.733799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:03.733858 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:31:03.733870 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:31:03.760641 27676 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:03.761158 27676 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:03.764971 27676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b64d9aa9407e4bec8d3ece62ff3a3cf5: Generated new TSK 0
I20260929 03:31:03.765172 27676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:03.774925 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27596
W20260929 03:31:03.778709 27581 connection.cc:535] client connection to 127.0.0.1:33593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33593): Connection reset by peer (error 104)
2026-09-29T03:31:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:03 dist-test-slave-z2zg krb5kdc[27695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:03 dist-test-slave-z2zg krb5kdc[27695](info): set up 2 sockets
Sep 29 03:31:03 dist-test-slave-z2zg krb5kdc[27695](info): commencing operation
krb5kdc: starting...
W20260929 03:31:04.796950 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:04 dist-test-slave-z2zg krb5kdc[27695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:04.909734 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:04.988500 27708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:04.988734 27708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:04.988771 27708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:04.989892 27708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:04.989957 27708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:04.989984 27708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:04.990026 27708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:04.990056 27708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:04.991554 27708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:04.991814 27708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:04.992033 27708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:04.996587 27716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:04.996596 27713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:04.996629 27714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:04.996745 27708 server_base.cc:1034] running on GCE node
I20260929 03:31:04.997069 27708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:04.997391 27708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:04.998574 27708 hybrid_clock.cc:648] HybridClock initialized: now 1790652664998546 us; error 44 us; skew 500 ppm
Sep 29 03:31:04 dist-test-slave-z2zg krb5kdc[27695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:05.000898 27708 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:05.001312 27708 webserver.cc:460] Webserver started at http://127.0.0.1:39353/ using document root <none> and password file <none>
I20260929 03:31:05.001542 27708 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:05.001610 27708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:05.001766 27708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:05.002758 27708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "a4d2f33e55fe44b88672ada03ae4d1ec"
format_stamp: "Formatted at 2026-09-29 03:31:04 on dist-test-slave-z2zg"
server_key: "0f208517a7dc368c4f127a8537badb24"
server_key_iv: "9696a3152e19cc2bed8d80d82f8a951e"
server_key_version: "encryptionkey@0"
I20260929 03:31:05.003171 27708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "a4d2f33e55fe44b88672ada03ae4d1ec"
format_stamp: "Formatted at 2026-09-29 03:31:04 on dist-test-slave-z2zg"
server_key: "0f208517a7dc368c4f127a8537badb24"
server_key_iv: "9696a3152e19cc2bed8d80d82f8a951e"
server_key_version: "encryptionkey@0"
I20260929 03:31:05.004456 27708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:31:05.005157 27723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:05.005329 27708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:31:05.005427 27708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "a4d2f33e55fe44b88672ada03ae4d1ec"
format_stamp: "Formatted at 2026-09-29 03:31:04 on dist-test-slave-z2zg"
server_key: "0f208517a7dc368c4f127a8537badb24"
server_key_iv: "9696a3152e19cc2bed8d80d82f8a951e"
server_key_version: "encryptionkey@0"
I20260929 03:31:05.005509 27708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:05.008736 27708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:05.010527 27708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:05.010691 27708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:05.014622 27708 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40593
I20260929 03:31:05.014664 27775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40593 every 8 connection(s)
I20260929 03:31:05.015033 27708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:05.016129 27776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:05.018230 27776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec: Bootstrap starting.
I20260929 03:31:05.018783 27776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:05.019119 27776 log.cc:824] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:05.019806 27776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec: No bootstrap required, opened a new log
I20260929 03:31:05.021341 27776 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260929 03:31:05.021466 27776 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:05.021487 27776 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d2f33e55fe44b88672ada03ae4d1ec, State: Initialized, Role: FOLLOWER
I20260929 03:31:05.021651 27776 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260929 03:31:05.021709 27776 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:05.021730 27776 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:05.021797 27776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:05.022317 27776 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260929 03:31:05.022400 27776 leader_election.cc:302] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d2f33e55fe44b88672ada03ae4d1ec; no voters:
I20260929 03:31:05.022548 27776 leader_election.cc:288] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:05.022629 27779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:05.022778 27779 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [term 1 LEADER]: Becoming Leader. State: Replica: a4d2f33e55fe44b88672ada03ae4d1ec, State: Running, Role: LEADER
I20260929 03:31:05.022806 27776 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:05.022900 27779 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } }
I20260929 03:31:05.023339 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d2f33e55fe44b88672ada03ae4d1ec. Latest consensus state: current_term: 1 leader_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } } }
I20260929 03:31:05.023428 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:05.023439 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d2f33e55fe44b88672ada03ae4d1ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40593 } } }
I20260929 03:31:05.023584 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:05.023921 27788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:05.023948 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27708
I20260929 03:31:05.024053 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:05.024251 6610 external_mini_cluster.cc:1428] Setting key 250aaf3d8df61ca6653850af1d90f10e
I20260929 03:31:05.024453 27788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:05.026232 27788 catalog_manager.cc:1347] Generated new cluster ID: dd8a3def82ba449fa20e0cd39ee3fa85
I20260929 03:31:05.026289 27788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:05.026800 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:05.024701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58749 (local address 127.0.0.1:40593)
0929 03:31:05.025103 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:05.025111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:05.025307 (+ 196us) server_negotiation.cc:407] Connection header received
0929 03:31:05.025460 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:05.025467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:05.025498 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:05.025650 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:05.026363 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:05.026366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:05.026367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:05.026370 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:05.026407 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:05.026428 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:05.026503 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:05.026579 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 03:31:05.026623 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":80,"threads_started":1}
I20260929 03:31:05.027635 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:05.030036 27797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:05.028933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:47375 (local address 127.0.0.1:40593)
0929 03:31:05.028970 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:05.028972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:05.029096 (+ 124us) server_negotiation.cc:407] Connection header received
0929 03:31:05.029166 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:05.029167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:05.029171 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:05.029226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:05.029800 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:05.029802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:05.029802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:05.029806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:05.029819 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:05.029823 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:05.029880 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:05.029934 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:31:05.029947 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:31:05.039947 27788 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:05.040107 27788 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:05.043752 27788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d2f33e55fe44b88672ada03ae4d1ec: Generated new TSK 0
I20260929 03:31:05.043910 27788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:05.048408 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27708
2026-09-29T03:31:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:05 dist-test-slave-z2zg krb5kdc[27807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:05 dist-test-slave-z2zg krb5kdc[27807](info): set up 2 sockets
Sep 29 03:31:05 dist-test-slave-z2zg krb5kdc[27807](info): commencing operation
krb5kdc: starting...
W20260929 03:31:06.069229 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:06.178362 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:06.257130 27820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:06.257318 27820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:06.257339 27820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:06.258533 27820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:06.258576 27820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:06.258591 27820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:06.258603 27820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:06.258616 27820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:06.260089 27820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:06.260267 27820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:06.260417 27820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:06.264940 27825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:06.264940 27826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:06.264915 27828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:06.265129 27820 server_base.cc:1034] running on GCE node
I20260929 03:31:06.265383 27820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:06.265715 27820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:06.266888 27820 hybrid_clock.cc:648] HybridClock initialized: now 1790652666266877 us; error 27 us; skew 500 ppm
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:06.272703 27820 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:06.273156 27820 webserver.cc:460] Webserver started at http://127.0.0.1:32917/ using document root <none> and password file <none>
I20260929 03:31:06.273379 27820 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:06.273447 27820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:06.273604 27820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:06.274585 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "dd0f5ff7fa964b3da90926f461ac8270"
format_stamp: "Formatted at 2026-09-29 03:31:06 on dist-test-slave-z2zg"
server_key: "d97c9d1fc4c2cfe63166b3220b470fe0"
server_key_iv: "8d9f54f5a9b8ac79355e03fae076358a"
server_key_version: "encryptionkey@0"
I20260929 03:31:06.274948 27820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "dd0f5ff7fa964b3da90926f461ac8270"
format_stamp: "Formatted at 2026-09-29 03:31:06 on dist-test-slave-z2zg"
server_key: "d97c9d1fc4c2cfe63166b3220b470fe0"
server_key_iv: "8d9f54f5a9b8ac79355e03fae076358a"
server_key_version: "encryptionkey@0"
I20260929 03:31:06.276229 27820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:06.276913 27835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:06.277102 27820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:06.277199 27820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "dd0f5ff7fa964b3da90926f461ac8270"
format_stamp: "Formatted at 2026-09-29 03:31:06 on dist-test-slave-z2zg"
server_key: "d97c9d1fc4c2cfe63166b3220b470fe0"
server_key_iv: "8d9f54f5a9b8ac79355e03fae076358a"
server_key_version: "encryptionkey@0"
I20260929 03:31:06.277277 27820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:06.287727 27820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:06.289919 27820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:06.290078 27820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:06.294111 27820 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33055
I20260929 03:31:06.294142 27887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33055 every 8 connection(s)
I20260929 03:31:06.294490 27820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:06.295426 27888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:06.297505 27888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: Bootstrap starting.
I20260929 03:31:06.298066 27888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:06.298344 27888 log.cc:824] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:06.298975 27888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: No bootstrap required, opened a new log
I20260929 03:31:06.300510 27888 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260929 03:31:06.300637 27888 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:06.300659 27888 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f5ff7fa964b3da90926f461ac8270, State: Initialized, Role: FOLLOWER
I20260929 03:31:06.300793 27888 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260929 03:31:06.300868 27888 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:06.300890 27888 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:06.300917 27888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:06.301371 27888 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260929 03:31:06.301453 27888 leader_election.cc:302] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0f5ff7fa964b3da90926f461ac8270; no voters:
I20260929 03:31:06.301563 27888 leader_election.cc:288] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:06.301626 27891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:06.301827 27891 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [term 1 LEADER]: Becoming Leader. State: Replica: dd0f5ff7fa964b3da90926f461ac8270, State: Running, Role: LEADER
I20260929 03:31:06.301836 27888 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:06.301923 27891 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } }
I20260929 03:31:06.302353 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27820
I20260929 03:31:06.302484 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:06.302678 6610 external_mini_cluster.cc:1428] Setting key f356b735eee8e5cc1b4c9908216d25ca
I20260929 03:31:06.303524 27892 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0f5ff7fa964b3da90926f461ac8270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } } }
I20260929 03:31:06.303666 27893 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0f5ff7fa964b3da90926f461ac8270. Latest consensus state: current_term: 1 leader_uuid: "dd0f5ff7fa964b3da90926f461ac8270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f5ff7fa964b3da90926f461ac8270" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33055 } } }
I20260929 03:31:06.303776 27893 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:06.303689 27892 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270 [sys.catalog]: This master's current role is: LEADER
W20260929 03:31:06.303875 27908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:06.303937 27908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:06.305145 27910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:06.305311 27910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:06.306825 27910 catalog_manager.cc:1347] Generated new cluster ID: 18f5c41b463048838d7b6011b89777a3
I20260929 03:31:06.306879 27910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 03:31:06.313320 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:06.303162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:33889 (local address 127.0.0.1:33055)
0929 03:31:06.303554 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:06.303565 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:06.303820 (+ 255us) server_negotiation.cc:407] Connection header received
0929 03:31:06.304039 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:06.304047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:06.304102 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:06.304204 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:06.305085 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:06.306279 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:06.307393 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:06.307792 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:06.307915 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:06.310124 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:06.310128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:06.310128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:06.310138 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:06.311798 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:06.312198 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:06.312203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:06.312203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:06.312296 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:06.312535 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:06.312535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:06.312536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:06.312764 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:06.312933 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:06.313015 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 03:31:06.313117 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":74,"threads_started":1}
I20260929 03:31:06.314284 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:06.329566 27906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:06.323321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:53069 (local address 127.0.0.1:33055)
0929 03:31:06.323359 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:06.323361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:06.323505 (+ 144us) server_negotiation.cc:407] Connection header received
0929 03:31:06.323575 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:06.323576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:06.323586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:06.323638 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:06.324154 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:06.324618 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:06.325277 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:06.325868 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:06.325948 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:06.327076 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:06.327077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:06.327078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:06.327084 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:06.328422 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:06.328788 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:06.328789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:06.328789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:06.328844 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:06.329064 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:06.329065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:06.329065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:06.329170 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:06.329287 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:06.329400 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:31:06.329456 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:31:06.346006 27910 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:06.346491 27910 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:06.350414 27910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0f5ff7fa964b3da90926f461ac8270: Generated new TSK 0
I20260929 03:31:06.350605 27910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:06.358839 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27820
W20260929 03:31:06.362699 27805 connection.cc:535] client connection to 127.0.0.1:33055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33055): Connection reset by peer (error 104)
2026-09-29T03:31:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27919](info): set up 2 sockets
Sep 29 03:31:06 dist-test-slave-z2zg krb5kdc[27919](info): commencing operation
krb5kdc: starting...
W20260929 03:31:07.376848 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:07 dist-test-slave-z2zg krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:07.490681 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:07.568797 27932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:07.569018 27932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:07.569056 27932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:07.570212 27932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:07.570281 27932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:07.570304 27932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:07.570333 27932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:07.570358 27932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:07.571861 27932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:07.572124 27932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:07.572338 27932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:07.576993 27940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:07.577016 27938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:07.577016 27937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:07.577116 27932 server_base.cc:1034] running on GCE node
I20260929 03:31:07.577442 27932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:07.577713 27932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:07.578851 27932 hybrid_clock.cc:648] HybridClock initialized: now 1790652667578843 us; error 30 us; skew 500 ppm
Sep 29 03:31:07 dist-test-slave-z2zg krb5kdc[27919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:07.581127 27932 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:07.581559 27932 webserver.cc:460] Webserver started at http://127.0.0.1:37361/ using document root <none> and password file <none>
I20260929 03:31:07.581780 27932 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:07.581845 27932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:07.581990 27932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:07.582937 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "f3f5ac09969449f29963dda500739763"
format_stamp: "Formatted at 2026-09-29 03:31:07 on dist-test-slave-z2zg"
server_key: "0a36f785f72d150cf062b4474d58f980"
server_key_iv: "44cacf4d9123230f3e0159d6bcb9e06a"
server_key_version: "encryptionkey@0"
I20260929 03:31:07.583323 27932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "f3f5ac09969449f29963dda500739763"
format_stamp: "Formatted at 2026-09-29 03:31:07 on dist-test-slave-z2zg"
server_key: "0a36f785f72d150cf062b4474d58f980"
server_key_iv: "44cacf4d9123230f3e0159d6bcb9e06a"
server_key_version: "encryptionkey@0"
I20260929 03:31:07.584606 27932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:31:07.585287 27947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:07.585484 27932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:31:07.585600 27932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "f3f5ac09969449f29963dda500739763"
format_stamp: "Formatted at 2026-09-29 03:31:07 on dist-test-slave-z2zg"
server_key: "0a36f785f72d150cf062b4474d58f980"
server_key_iv: "44cacf4d9123230f3e0159d6bcb9e06a"
server_key_version: "encryptionkey@0"
I20260929 03:31:07.585688 27932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:07.606326 27932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:07.607985 27932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:07.608151 27932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:07.611912 27932 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41339
I20260929 03:31:07.611948 27999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41339 every 8 connection(s)
I20260929 03:31:07.612257 27932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:07.613039 28000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:07.615236 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 27932
I20260929 03:31:07.615249 28000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: Bootstrap starting.
I20260929 03:31:07.615356 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:07.615533 6610 external_mini_cluster.cc:1428] Setting key 201cddafdd073f26da489e6d6772d3aa
I20260929 03:31:07.615875 28000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:07.616245 28000 log.cc:824] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:07.616922 28000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: No bootstrap required, opened a new log
I20260929 03:31:07.618901 28000 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 03:31:07.619087 28000 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:07.619125 28000 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f5ac09969449f29963dda500739763, State: Initialized, Role: FOLLOWER
I20260929 03:31:07.619261 28000 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 03:31:07.619343 28000 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:07.619366 28000 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:07.619431 28000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:07.620092 28000 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 03:31:07.620182 28000 leader_election.cc:302] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f5ac09969449f29963dda500739763; no voters:
I20260929 03:31:07.620347 28000 leader_election.cc:288] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:07.620368 28005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:07.620466 28005 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [term 1 LEADER]: Becoming Leader. State: Replica: f3f5ac09969449f29963dda500739763, State: Running, Role: LEADER
I20260929 03:31:07.620584 28000 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:07.620559 28005 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } }
I20260929 03:31:07.620913 28005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f5ac09969449f29963dda500739763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } } }
I20260929 03:31:07.621037 28005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:07.621217 28005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f5ac09969449f29963dda500739763. Latest consensus state: current_term: 1 leader_uuid: "f3f5ac09969449f29963dda500739763" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5ac09969449f29963dda500739763" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41339 } } }
I20260929 03:31:07.621332 28005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:07.621556 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:07.616014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:58809 (local address 127.0.0.1:41339)
0929 03:31:07.616168 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:07.616176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:07.616212 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:31:07.616772 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:07.616780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:07.616827 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:07.616936 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:07.617185 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:07.618009 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:07.619220 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:07.619718 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:07.619817 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:07.620523 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:07.620527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:07.620528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:07.620751 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:07.620786 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:07.620818 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:07.620896 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:07.621174 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 03:31:07.621202 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260929 03:31:07.622486 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 03:31:07.622622 28021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:07.622679 28021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:07.622723 28012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:07.622877 28012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:07.624411 28012 catalog_manager.cc:1347] Generated new cluster ID: d43fbf8371fd45ac933cbf0b9975024a
I20260929 03:31:07.624464 28012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:07.626754 28003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:07.624059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:59105 (local address 127.0.0.1:41339)
0929 03:31:07.624089 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:07.624090 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:07.624098 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:07.624116 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:07.624117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:07.624125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:07.624177 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:07.624414 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:07.624894 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:07.625605 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:07.625841 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:07.625924 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:07.626541 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:07.626542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:07.626542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:07.626547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:07.626559 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:07.626563 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:07.626616 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:07.626676 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:31:07.626684 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:31:07.651681 28012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:07.652367 28012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:07.660708 28012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f5ac09969449f29963dda500739763: Generated new TSK 0
I20260929 03:31:07.660885 28012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:07.669122 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 27932
W20260929 03:31:07.673554 27917 connection.cc:535] client connection to 127.0.0.1:41339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41339): Connection reset by peer (error 104)
2026-09-29T03:31:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:07 dist-test-slave-z2zg krb5kdc[28031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:07 dist-test-slave-z2zg krb5kdc[28031](info): set up 2 sockets
Sep 29 03:31:07 dist-test-slave-z2zg krb5kdc[28031](info): commencing operation
krb5kdc: starting...
W20260929 03:31:08.688975 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:08 dist-test-slave-z2zg krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 03:31:08.797646 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:08.875770 28044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:08.875954 28044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:08.875975 28044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:08.877094 28044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:08.877134 28044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:08.877148 28044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:08.877209 28044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:08.877228 28044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:08.878646 28044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:08.878888 28044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:08.879161 28044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:08.883774 28050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:08.883870 28052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:08.883813 28049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:08.884018 28044 server_base.cc:1034] running on GCE node
I20260929 03:31:08.884280 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:08.884562 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:08.885735 28044 hybrid_clock.cc:648] HybridClock initialized: now 1790652668885724 us; error 35 us; skew 500 ppm
Sep 29 03:31:08 dist-test-slave-z2zg krb5kdc[28031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:08.888232 28044 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 03:31:08.888587 28044 webserver.cc:460] Webserver started at http://127.0.0.1:41485/ using document root <none> and password file <none>
I20260929 03:31:08.888770 28044 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:08.888813 28044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:08.888895 28044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:08.889812 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "d4a7301f784b41ad959811fee4b8c145"
format_stamp: "Formatted at 2026-09-29 03:31:08 on dist-test-slave-z2zg"
server_key: "c695fea08de022f4ad1fd85cc3d0f67a"
server_key_iv: "c7e76706d44042f051f5f1de229b9f8b"
server_key_version: "encryptionkey@0"
I20260929 03:31:08.890092 28044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "d4a7301f784b41ad959811fee4b8c145"
format_stamp: "Formatted at 2026-09-29 03:31:08 on dist-test-slave-z2zg"
server_key: "c695fea08de022f4ad1fd85cc3d0f67a"
server_key_iv: "c7e76706d44042f051f5f1de229b9f8b"
server_key_version: "encryptionkey@0"
I20260929 03:31:08.891332 28044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:31:08.891996 28059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:08.892169 28044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:08.892251 28044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "d4a7301f784b41ad959811fee4b8c145"
format_stamp: "Formatted at 2026-09-29 03:31:08 on dist-test-slave-z2zg"
server_key: "c695fea08de022f4ad1fd85cc3d0f67a"
server_key_iv: "c7e76706d44042f051f5f1de229b9f8b"
server_key_version: "encryptionkey@0"
I20260929 03:31:08.892305 28044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:08.895381 28044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:08.897087 28044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:08.897186 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:08.900748 28044 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39629
I20260929 03:31:08.900775 28111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39629 every 8 connection(s)
I20260929 03:31:08.901082 28044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:08.901374 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28044
I20260929 03:31:08.901484 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:08.901682 6610 external_mini_cluster.cc:1428] Setting key ecbfd48aa7ca08de8735f276e9fadc50
I20260929 03:31:08.902261 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:08.903975 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:08.902149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:51371 (local address 127.0.0.1:39629)
0929 03:31:08.902327 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:08.902336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:08.902365 (+ 29us) server_negotiation.cc:407] Connection header received
0929 03:31:08.902756 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:08.902762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:08.902783 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:08.902898 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:08.903545 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:08.903548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:08.903548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:08.903552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:08.903587 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:08.903597 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:08.903657 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:08.903742 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 03:31:08.903793 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260929 03:31:08.904928 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:08.905167 28112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145: Bootstrap starting.
I20260929 03:31:08.905866 28112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:08.906209 28112 log.cc:824] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:08.906996 28112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145: No bootstrap required, opened a new log
I20260929 03:31:08.907521 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:08.906469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:46963 (local address 127.0.0.1:39629)
0929 03:31:08.906507 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:08.906509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:08.906664 (+ 155us) server_negotiation.cc:407] Connection header received
0929 03:31:08.906721 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:08.906721 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:08.906725 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:08.906780 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:08.907326 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:08.907327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:08.907327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:08.907331 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:08.907344 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:08.907348 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:08.907411 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:08.907444 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 03:31:08.907456 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:08.909056 28112 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } }
I20260929 03:31:08.909183 28112 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:08.909206 28112 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a7301f784b41ad959811fee4b8c145, State: Initialized, Role: FOLLOWER
I20260929 03:31:08.909281 28112 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } }
I20260929 03:31:08.909332 28112 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:08.909394 28112 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:08.909446 28112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:08.909947 28112 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } }
I20260929 03:31:08.910064 28112 leader_election.cc:302] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a7301f784b41ad959811fee4b8c145; no voters:
I20260929 03:31:08.910245 28112 leader_election.cc:288] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:08.910290 28122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:08.910460 28122 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [term 1 LEADER]: Becoming Leader. State: Replica: d4a7301f784b41ad959811fee4b8c145, State: Running, Role: LEADER
I20260929 03:31:08.910485 28112 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:08.910542 28122 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } }
I20260929 03:31:08.910861 28123 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a7301f784b41ad959811fee4b8c145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } } }
I20260929 03:31:08.910859 28124 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a7301f784b41ad959811fee4b8c145. Latest consensus state: current_term: 1 leader_uuid: "d4a7301f784b41ad959811fee4b8c145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a7301f784b41ad959811fee4b8c145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39629 } } }
I20260929 03:31:08.910966 28123 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:08.910974 28124 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:08.911237 28128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:08.911927 28128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:08.913415 28128 catalog_manager.cc:1347] Generated new cluster ID: 21e9df4fb6eb406299b1f883bf7955aa
I20260929 03:31:08.913467 28128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:08.929103 28128 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:08.929272 28128 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:08.934942 28128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a7301f784b41ad959811fee4b8c145: Generated new TSK 0
I20260929 03:31:08.935134 28128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:08.942162 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28044
2026-09-29T03:31:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:08 dist-test-slave-z2zg krb5kdc[28143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:08 dist-test-slave-z2zg krb5kdc[28143](info): set up 2 sockets
Sep 29 03:31:08 dist-test-slave-z2zg krb5kdc[28143](info): commencing operation
krb5kdc: starting...
W20260929 03:31:09.957290 6610 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:10.070237 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43077
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:10.148821 28156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:10.149047 28156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:10.149086 28156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:10.150235 28156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:10.150317 28156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:10.150344 28156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:10.150374 28156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:10.150425 28156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:10.152027 28156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:10.152297 28156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:10.152522 28156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:10.157192 28164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:10.157200 28162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:10.157332 28156 server_base.cc:1034] running on GCE node
W20260929 03:31:10.157203 28161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:10.157660 28156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:10.157941 28156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:10.159106 28156 hybrid_clock.cc:648] HybridClock initialized: now 1790652670159101 us; error 36 us; skew 500 ppm
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652670, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:10.161375 28156 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:10.161782 28156 webserver.cc:460] Webserver started at http://127.6.116.190:37257/ using document root <none> and password file <none>
I20260929 03:31:10.162011 28156 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:10.162076 28156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:10.162216 28156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:10.163264 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "87035406bae34766b8e4c5ed38eb047f"
format_stamp: "Formatted at 2026-09-29 03:31:10 on dist-test-slave-z2zg"
server_key: "cf19b9856020828d9f0c96819e0ee57e"
server_key_iv: "4b8e82bc0b9ac62d6fe6f0b9c16c0532"
server_key_version: "encryptionkey@0"
I20260929 03:31:10.163612 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "87035406bae34766b8e4c5ed38eb047f"
format_stamp: "Formatted at 2026-09-29 03:31:10 on dist-test-slave-z2zg"
server_key: "cf19b9856020828d9f0c96819e0ee57e"
server_key_iv: "4b8e82bc0b9ac62d6fe6f0b9c16c0532"
server_key_version: "encryptionkey@0"
I20260929 03:31:10.164875 28156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:31:10.165529 28171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:10.165702 28156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:31:10.165797 28156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "87035406bae34766b8e4c5ed38eb047f"
format_stamp: "Formatted at 2026-09-29 03:31:10 on dist-test-slave-z2zg"
server_key: "cf19b9856020828d9f0c96819e0ee57e"
server_key_iv: "4b8e82bc0b9ac62d6fe6f0b9c16c0532"
server_key_version: "encryptionkey@0"
I20260929 03:31:10.165874 28156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:10.195953 28156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:10.197772 28156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:10.197944 28156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:10.201643 28156 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:43077
I20260929 03:31:10.201690 28223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:43077 every 8 connection(s)
I20260929 03:31:10.202033 28156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:10.202864 28224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:10.204313 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28156
I20260929 03:31:10.204442 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:10.204612 6610 external_mini_cluster.cc:1428] Setting key e53393af4a0aa8a7b526bcabb424cf54
I20260929 03:31:10.205444 28224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f: Bootstrap starting.
I20260929 03:31:10.206117 28224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:10.206387 28224 log.cc:824] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:10.207114 28224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f: No bootstrap required, opened a new log
I20260929 03:31:10.209127 28224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } }
I20260929 03:31:10.209287 28224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:10.209324 28224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87035406bae34766b8e4c5ed38eb047f, State: Initialized, Role: FOLLOWER
I20260929 03:31:10.209499 28224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } }
I20260929 03:31:10.209600 28224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:10.209642 28224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:10.209712 28224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:10.210462 28224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } }
I20260929 03:31:10.210583 28224 leader_election.cc:302] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87035406bae34766b8e4c5ed38eb047f; no voters:
I20260929 03:31:10.210734 28224 leader_election.cc:288] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:10.210767 28229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:10.210943 28229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [term 1 LEADER]: Becoming Leader. State: Replica: 87035406bae34766b8e4c5ed38eb047f, State: Running, Role: LEADER
I20260929 03:31:10.210990 28224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:10.211086 28229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } }
I20260929 03:31:10.211365 28229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87035406bae34766b8e4c5ed38eb047f. Latest consensus state: current_term: 1 leader_uuid: "87035406bae34766b8e4c5ed38eb047f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } } }
I20260929 03:31:10.211455 28229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [sys.catalog]: This master's current role is: LEADER
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:10.211376 28230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87035406bae34766b8e4c5ed38eb047f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87035406bae34766b8e4c5ed38eb047f" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43077 } } }
I20260929 03:31:10.211615 28230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:10.212693 28235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:10.213239 28235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:10.214727 28235 catalog_manager.cc:1347] Generated new cluster ID: 6359ae390718458799152f2f2ad055fa
I20260929 03:31:10.214775 28235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:10.215731 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:10.205114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52279 (local address 127.6.116.190:43077)
0929 03:31:10.205299 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:10.205308 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:10.205338 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:31:10.205887 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:10.205895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:10.205951 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:10.206108 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:10.206756 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:10.207752 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:10.208816 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:10.209174 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:10.209246 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:10.212166 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:10.212170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:10.212170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:10.212182 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:10.214023 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:10.214475 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:10.214478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:10.214478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:10.214587 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:10.214825 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:10.214826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:10.214826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:10.215134 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:10.215282 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:10.215403 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 03:31:10.215499 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
I20260929 03:31:10.216604 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:10.225114 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:10.217915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40619 (local address 127.6.116.190:43077)
0929 03:31:10.217948 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:10.217950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:10.218073 (+ 123us) server_negotiation.cc:407] Connection header received
0929 03:31:10.218114 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:10.218115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:10.218125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:10.218189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:10.218707 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:10.219791 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:10.220731 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:10.221088 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:10.221155 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:10.222393 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:10.222394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:10.222395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:10.222401 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:10.223984 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:10.224353 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:10.224354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:10.224354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:10.224408 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:10.224641 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:10.224642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:10.224643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:10.224760 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:10.224858 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:10.224972 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 03:31:10.225031 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:10.228101 28235 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:10.228546 28235 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:10.232690 28235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87035406bae34766b8e4c5ed38eb047f: Generated new TSK 0
I20260929 03:31:10.232851 28235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:10.238847 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28156
W20260929 03:31:10.242933 28141 connection.cc:535] client connection to 127.6.116.190:43077 recv error: Network error: failed to read from TLS socket (remote: 127.6.116.190:43077): Connection reset by peer (error 104)
2026-09-29T03:31:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28255](info): set up 2 sockets
Sep 29 03:31:10 dist-test-slave-z2zg krb5kdc[28255](info): commencing operation
krb5kdc: starting...
W20260929 03:31:11.268692 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:11 dist-test-slave-z2zg krb5kdc[28255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:11.378432 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:33165
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:33165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:11.456437 28268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:11.456646 28268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:11.456683 28268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:11.457803 28268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:11.457870 28268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:11.457896 28268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:11.457926 28268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:11.457955 28268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:11.459455 28268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:33165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:11.459735 28268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:11.459946 28268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:11.464639 28276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:11.464669 28273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:11.464619 28274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:11.464913 28268 server_base.cc:1034] running on GCE node
I20260929 03:31:11.465154 28268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:11.465421 28268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:11.466588 28268 hybrid_clock.cc:648] HybridClock initialized: now 1790652671466579 us; error 39 us; skew 500 ppm
Sep 29 03:31:11 dist-test-slave-z2zg krb5kdc[28255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652671, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:11.472232 28268 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:11.472681 28268 webserver.cc:460] Webserver started at http://127.6.116.190:45949/ using document root <none> and password file <none>
I20260929 03:31:11.472903 28268 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:11.472968 28268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:11.473119 28268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:11.474082 28268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "bcc71e50381e42ec86c3ce3e69797091"
format_stamp: "Formatted at 2026-09-29 03:31:11 on dist-test-slave-z2zg"
server_key: "f2be9bfcdb9cad79620d22a88b2e142e"
server_key_iv: "800058dc3a5b26bd318233e33bafc3e5"
server_key_version: "encryptionkey@0"
I20260929 03:31:11.474418 28268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "bcc71e50381e42ec86c3ce3e69797091"
format_stamp: "Formatted at 2026-09-29 03:31:11 on dist-test-slave-z2zg"
server_key: "f2be9bfcdb9cad79620d22a88b2e142e"
server_key_iv: "800058dc3a5b26bd318233e33bafc3e5"
server_key_version: "encryptionkey@0"
I20260929 03:31:11.475778 28268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 03:31:11.476466 28283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:11.476639 28268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:11.476737 28268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "bcc71e50381e42ec86c3ce3e69797091"
format_stamp: "Formatted at 2026-09-29 03:31:11 on dist-test-slave-z2zg"
server_key: "f2be9bfcdb9cad79620d22a88b2e142e"
server_key_iv: "800058dc3a5b26bd318233e33bafc3e5"
server_key_version: "encryptionkey@0"
I20260929 03:31:11.476806 28268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:11.492050 28268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:11.494172 28268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:11.494329 28268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:11.498158 28268 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:33165
I20260929 03:31:11.498189 28335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:33165 every 8 connection(s)
I20260929 03:31:11.498505 28268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:11.499303 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:11.501368 28336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091: Bootstrap starting.
I20260929 03:31:11.501848 28336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:11.502065 28336 log.cc:824] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:11.502321 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28268
I20260929 03:31:11.502424 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:11.502615 6610 external_mini_cluster.cc:1428] Setting key d894b1d6f1b6875348270882a1043e04
I20260929 03:31:11.502839 28336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091: No bootstrap required, opened a new log
I20260929 03:31:11.504495 28336 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } }
I20260929 03:31:11.504645 28336 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:11.504707 28336 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc71e50381e42ec86c3ce3e69797091, State: Initialized, Role: FOLLOWER
I20260929 03:31:11.504829 28336 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } }
I20260929 03:31:11.504923 28336 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:11.504966 28336 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:11.505028 28336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:11.505522 28336 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } }
I20260929 03:31:11.505646 28336 leader_election.cc:302] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc71e50381e42ec86c3ce3e69797091; no voters:
I20260929 03:31:11.505827 28336 leader_election.cc:288] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:11.505852 28341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:11.506021 28341 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [term 1 LEADER]: Becoming Leader. State: Replica: bcc71e50381e42ec86c3ce3e69797091, State: Running, Role: LEADER
I20260929 03:31:11.506083 28336 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:11.506141 28341 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } }
I20260929 03:31:11.506429 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc71e50381e42ec86c3ce3e69797091. Latest consensus state: current_term: 1 leader_uuid: "bcc71e50381e42ec86c3ce3e69797091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } } }
I20260929 03:31:11.506558 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:11.506963 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc71e50381e42ec86c3ce3e69797091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc71e50381e42ec86c3ce3e69797091" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 33165 } } }
I20260929 03:31:11.507117 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:11.506995 28348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:11.507674 28348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:11.509183 28348 catalog_manager.cc:1347] Generated new cluster ID: ca6f5bf8096d4a5eb4179e0bd8fd511b
I20260929 03:31:11.509266 28348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:11.509543 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:11.503103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59497 (local address 127.6.116.190:33165)
0929 03:31:11.503296 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:11.503304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:11.503336 (+ 32us) server_negotiation.cc:407] Connection header received
0929 03:31:11.504479 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:11.504484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:11.504524 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:11.504656 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:11.504869 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:11.506318 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:11.507547 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:11.508124 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:11.508231 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:11.508899 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:11.508903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:11.508904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:11.509089 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:11.509124 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:11.509138 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:11.509225 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:11.509328 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:31:11.509356 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260929 03:31:11.510389 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:11.514989 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:11.511717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46115 (local address 127.6.116.190:33165)
0929 03:31:11.511745 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:11.511747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:11.511837 (+ 90us) server_negotiation.cc:407] Connection header received
0929 03:31:11.511893 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:11.511893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:11.511900 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:11.511952 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:11.512067 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:11.512750 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:11.513747 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:11.514054 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:11.514109 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:11.514754 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:11.514755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:11.514755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:11.514759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:11.514770 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:11.514774 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:11.514826 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:11.514917 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 03:31:11.514927 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:31:11.517838 28348 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:11.518307 28348 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:11.523156 28348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcc71e50381e42ec86c3ce3e69797091: Generated new TSK 0
I20260929 03:31:11.523325 28348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:11.528419 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28268
W20260929 03:31:11.532783 28253 connection.cc:535] client connection to 127.6.116.190:33165 recv error: Network error: failed to read from TLS socket (remote: 127.6.116.190:33165): Connection reset by peer (error 104)
2026-09-29T03:31:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:11 dist-test-slave-z2zg krb5kdc[28367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:11 dist-test-slave-z2zg krb5kdc[28367](info): set up 2 sockets
Sep 29 03:31:11 dist-test-slave-z2zg krb5kdc[28367](info): commencing operation
krb5kdc: starting...
W20260929 03:31:12.588927 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:12 dist-test-slave-z2zg krb5kdc[28367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:12.695879 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42407
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:42407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:12.774897 28380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:12.775148 28380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:12.775175 28380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:12.776247 28380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:12.776280 28380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:12.776294 28380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:12.776307 28380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:12.776319 28380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:12.777666 28380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:42407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:42407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:12.777837 28380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:12.777988 28380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:12.782768 28385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:12.782769 28388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:12.782792 28386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:12.782784 28380 server_base.cc:1034] running on GCE node
I20260929 03:31:12.783267 28380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:12.783569 28380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:12.784734 28380 hybrid_clock.cc:648] HybridClock initialized: now 1790652672784718 us; error 48 us; skew 500 ppm
Sep 29 03:31:12 dist-test-slave-z2zg krb5kdc[28367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652672, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:12.787194 28380 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:12.787650 28380 webserver.cc:460] Webserver started at http://127.6.116.190:42833/ using document root <none> and password file <none>
I20260929 03:31:12.787896 28380 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:12.787966 28380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:12.788102 28380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:12.789108 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "d0d048d33d094df0af1d3682ad0b39c5"
format_stamp: "Formatted at 2026-09-29 03:31:12 on dist-test-slave-z2zg"
server_key: "f5001e651187cf37efef0bcdad2b8539"
server_key_iv: "86003b6a0b8bad4ce3c630ecd48f5940"
server_key_version: "encryptionkey@0"
I20260929 03:31:12.789469 28380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "d0d048d33d094df0af1d3682ad0b39c5"
format_stamp: "Formatted at 2026-09-29 03:31:12 on dist-test-slave-z2zg"
server_key: "f5001e651187cf37efef0bcdad2b8539"
server_key_iv: "86003b6a0b8bad4ce3c630ecd48f5940"
server_key_version: "encryptionkey@0"
I20260929 03:31:12.790751 28380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:31:12.791553 28395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:12.791718 28380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:12.791812 28380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "d0d048d33d094df0af1d3682ad0b39c5"
format_stamp: "Formatted at 2026-09-29 03:31:12 on dist-test-slave-z2zg"
server_key: "f5001e651187cf37efef0bcdad2b8539"
server_key_iv: "86003b6a0b8bad4ce3c630ecd48f5940"
server_key_version: "encryptionkey@0"
I20260929 03:31:12.791890 28380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:12.795022 28380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:12.796893 28380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:12.797044 28380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:12.801074 28380 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:42407
I20260929 03:31:12.801101 28447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:42407 every 8 connection(s)
I20260929 03:31:12.801409 28380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:12.802363 28448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:12.804600 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5: Bootstrap starting.
I20260929 03:31:12.805145 28448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:12.805425 28448 log.cc:824] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:12.806180 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5: No bootstrap required, opened a new log
I20260929 03:31:12.807807 28448 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } }
I20260929 03:31:12.807946 28448 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:12.807968 28448 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d048d33d094df0af1d3682ad0b39c5, State: Initialized, Role: FOLLOWER
I20260929 03:31:12.808122 28448 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } }
I20260929 03:31:12.808179 28448 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:12.808199 28448 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:12.808245 28448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:12.809080 28448 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } }
I20260929 03:31:12.809163 28448 leader_election.cc:302] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0d048d33d094df0af1d3682ad0b39c5; no voters:
I20260929 03:31:12.809346 28448 leader_election.cc:288] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:12.809489 28452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:12.809607 28448 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:12.809680 28452 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [term 1 LEADER]: Becoming Leader. State: Replica: d0d048d33d094df0af1d3682ad0b39c5, State: Running, Role: LEADER
I20260929 03:31:12.809844 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28380
I20260929 03:31:12.809830 28452 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } }
I20260929 03:31:12.809952 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:12.810146 6610 external_mini_cluster.cc:1428] Setting key df2a344f3bade51dc5c521e78701af13
I20260929 03:31:12.810178 28451 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0d048d33d094df0af1d3682ad0b39c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } } }
I20260929 03:31:12.810261 28451 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:12.810415 28453 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0d048d33d094df0af1d3682ad0b39c5. Latest consensus state: current_term: 1 leader_uuid: "d0d048d33d094df0af1d3682ad0b39c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0d048d33d094df0af1d3682ad0b39c5" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 42407 } } }
I20260929 03:31:12.810474 28453 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:12.811257 28460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:12.811820 28460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:12.812604 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:12.810813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60695 (local address 127.6.116.190:42407)
0929 03:31:12.811015 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:12.811025 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:12.811079 (+ 54us) server_negotiation.cc:407] Connection header received
0929 03:31:12.811243 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:12.811251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:12.811282 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:12.811443 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:12.812149 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:12.812153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:12.812153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:12.812157 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:12.812211 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:12.812222 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:12.812296 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:12.812366 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 03:31:12.812407 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260929 03:31:12.813530 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:12.813546 28460 catalog_manager.cc:1347] Generated new cluster ID: b7f3188a9dab4c0c9c86b022c1b83860
I20260929 03:31:12.813581 28460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:12.816031 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:12.814968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39187 (local address 127.6.116.190:42407)
0929 03:31:12.814996 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:12.815005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:12.815157 (+ 152us) server_negotiation.cc:407] Connection header received
0929 03:31:12.815217 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:12.815218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:12.815231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:12.815285 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:12.815833 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:12.815834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:12.815834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:12.815838 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:12.815849 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:12.815853 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:12.815902 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:12.815962 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 03:31:12.815973 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:31:12.833830 28460 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:12.833984 28460 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:12.837764 28460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0d048d33d094df0af1d3682ad0b39c5: Generated new TSK 0
I20260929 03:31:12.837922 28460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:12.845191 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28380
2026-09-29T03:31:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:12 dist-test-slave-z2zg krb5kdc[28479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:12 dist-test-slave-z2zg krb5kdc[28479](info): set up 2 sockets
Sep 29 03:31:12 dist-test-slave-z2zg krb5kdc[28479](info): commencing operation
krb5kdc: starting...
W20260929 03:31:13.869017 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:13 dist-test-slave-z2zg krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:13.978140 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43113
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:43113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:14.055269 28492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:14.055492 28492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:14.055514 28492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:14.056694 28492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:14.056746 28492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:14.056761 28492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:14.056773 28492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:14.056787 28492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:14.058182 28492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:43113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:14.058365 28492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:14.058521 28492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:14.063140 28500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:14.063154 28497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:14.063270 28498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:14.063417 28492 server_base.cc:1034] running on GCE node
I20260929 03:31:14.063668 28492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:14.063946 28492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:14.065124 28492 hybrid_clock.cc:648] HybridClock initialized: now 1790652674065112 us; error 32 us; skew 500 ppm
Sep 29 03:31:14 dist-test-slave-z2zg krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652674, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:14.067648 28492 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:14.068096 28492 webserver.cc:460] Webserver started at http://127.6.116.190:39515/ using document root <none> and password file <none>
I20260929 03:31:14.068324 28492 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:14.068391 28492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:14.068535 28492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:14.069495 28492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "90494033cf5d4438bc1e74e5421cdf3e"
format_stamp: "Formatted at 2026-09-29 03:31:14 on dist-test-slave-z2zg"
server_key: "53b363bc1579b79295a94037e3146cf0"
server_key_iv: "7c7e8de9d7b84c2e749c985d09f6947e"
server_key_version: "encryptionkey@0"
I20260929 03:31:14.069844 28492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "90494033cf5d4438bc1e74e5421cdf3e"
format_stamp: "Formatted at 2026-09-29 03:31:14 on dist-test-slave-z2zg"
server_key: "53b363bc1579b79295a94037e3146cf0"
server_key_iv: "7c7e8de9d7b84c2e749c985d09f6947e"
server_key_version: "encryptionkey@0"
I20260929 03:31:14.071223 28492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 03:31:14.071974 28507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:14.072165 28492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:31:14.072258 28492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "90494033cf5d4438bc1e74e5421cdf3e"
format_stamp: "Formatted at 2026-09-29 03:31:14 on dist-test-slave-z2zg"
server_key: "53b363bc1579b79295a94037e3146cf0"
server_key_iv: "7c7e8de9d7b84c2e749c985d09f6947e"
server_key_version: "encryptionkey@0"
I20260929 03:31:14.072341 28492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:14.095181 28492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:14.096946 28492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:14.097115 28492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:14.101032 28492 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:43113
I20260929 03:31:14.101071 28559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:43113 every 8 connection(s)
I20260929 03:31:14.101432 28492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:14.102005 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28492
I20260929 03:31:14.102123 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:14.102310 6610 external_mini_cluster.cc:1428] Setting key 799949963f539db8bf836a1dc93e46da
I20260929 03:31:14.102360 28560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:14.105726 28560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e: Bootstrap starting.
I20260929 03:31:14.106328 28560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:14.106660 28560 log.cc:824] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:14.107442 28560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e: No bootstrap required, opened a new log
Sep 29 03:31:14 dist-test-slave-z2zg krb5kdc[28479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:14.109477 28560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } }
I20260929 03:31:14.109647 28560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:14.109691 28560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90494033cf5d4438bc1e74e5421cdf3e, State: Initialized, Role: FOLLOWER
I20260929 03:31:14.109831 28560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } }
I20260929 03:31:14.109918 28560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:14.109958 28560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:14.110015 28560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:14.110553 28560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } }
I20260929 03:31:14.110673 28560 leader_election.cc:302] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90494033cf5d4438bc1e74e5421cdf3e; no voters:
I20260929 03:31:14.110864 28560 leader_election.cc:288] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:14.110895 28565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:14.111007 28565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [term 1 LEADER]: Becoming Leader. State: Replica: 90494033cf5d4438bc1e74e5421cdf3e, State: Running, Role: LEADER
I20260929 03:31:14.111145 28560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:14.111125 28565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } }
I20260929 03:31:14.111477 28565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90494033cf5d4438bc1e74e5421cdf3e. Latest consensus state: current_term: 1 leader_uuid: "90494033cf5d4438bc1e74e5421cdf3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } } }
I20260929 03:31:14.111616 28565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:14.112051 28573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:14.112470 28566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90494033cf5d4438bc1e74e5421cdf3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90494033cf5d4438bc1e74e5421cdf3e" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 43113 } } }
I20260929 03:31:14.112586 28573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:14.112607 28566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:14.113636 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:14.102773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51633 (local address 127.6.116.190:43113)
0929 03:31:14.102953 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:14.102963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:14.102997 (+ 34us) server_negotiation.cc:407] Connection header received
0929 03:31:14.103583 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:14.103592 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:14.103661 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:14.103814 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:14.104383 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:14.105745 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:14.106532 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:14.107258 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:14.107375 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:14.109999 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:14.110003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:14.110003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:14.110014 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:14.112032 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:14.112449 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:14.112452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:14.112452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:14.112569 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:14.112788 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:14.112789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:14.112789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:14.113079 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:14.113223 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:14.113332 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:31:14.113465 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260929 03:31:14.114521 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:14.114671 28573 catalog_manager.cc:1347] Generated new cluster ID: 4f649d30797c440589c979e431b92588
I20260929 03:31:14.114728 28573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:14.121690 28563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:14.115913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57845 (local address 127.6.116.190:43113)
0929 03:31:14.115944 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:14.115946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:14.116047 (+ 101us) server_negotiation.cc:407] Connection header received
0929 03:31:14.116107 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:14.116107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:14.116116 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:14.116172 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:14.116657 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:14.117109 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:14.117780 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:14.117992 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:14.118082 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:14.119230 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:14.119231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:14.119232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:14.119238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:14.120583 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:14.120932 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:14.120933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:14.120934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:14.120984 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:14.121191 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:14.121192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:14.121192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:14.121298 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:14.121392 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:14.121501 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 03:31:14.121580 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:14.134532 28573 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:14.135251 28573 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:14.145367 28573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90494033cf5d4438bc1e74e5421cdf3e: Generated new TSK 0
I20260929 03:31:14.145527 28573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:14.154016 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28492
2026-09-29T03:31:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:14 dist-test-slave-z2zg krb5kdc[28591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:14 dist-test-slave-z2zg krb5kdc[28591](info): set up 2 sockets
Sep 29 03:31:14 dist-test-slave-z2zg krb5kdc[28591](info): commencing operation
krb5kdc: starting...
W20260929 03:31:15.180540 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:15 dist-test-slave-z2zg krb5kdc[28591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:15.287892 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38135
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:45715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:15.365324 28604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:15.365497 28604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:15.365518 28604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:15.366627 28604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:15.366667 28604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:15.366681 28604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:15.366735 28604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:15.366763 28604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:15.368178 28604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:45715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:15.368384 28604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:15.368594 28604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:15.373099 28610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:15.373164 28609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:15.373273 28604 server_base.cc:1034] running on GCE node
W20260929 03:31:15.373301 28612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:15.373525 28604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:15.373813 28604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:15.374979 28604 hybrid_clock.cc:648] HybridClock initialized: now 1790652675374963 us; error 28 us; skew 500 ppm
Sep 29 03:31:15 dist-test-slave-z2zg krb5kdc[28591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:15.377413 28604 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:15.377866 28604 webserver.cc:460] Webserver started at http://127.6.116.190:33279/ using document root <none> and password file <none>
I20260929 03:31:15.378098 28604 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:15.378168 28604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:15.378316 28604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:15.379369 28604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "31b7be69a15e41f4a8e2add292479265"
format_stamp: "Formatted at 2026-09-29 03:31:15 on dist-test-slave-z2zg"
server_key: "4f54fc5c26e639741ecdaae6605c13e9"
server_key_iv: "6bcf785198aa1d718d24bbfdc2041159"
server_key_version: "encryptionkey@0"
I20260929 03:31:15.379722 28604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "31b7be69a15e41f4a8e2add292479265"
format_stamp: "Formatted at 2026-09-29 03:31:15 on dist-test-slave-z2zg"
server_key: "4f54fc5c26e639741ecdaae6605c13e9"
server_key_iv: "6bcf785198aa1d718d24bbfdc2041159"
server_key_version: "encryptionkey@0"
I20260929 03:31:15.380977 28604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:31:15.381731 28619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:15.381929 28604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:31:15.382033 28604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "31b7be69a15e41f4a8e2add292479265"
format_stamp: "Formatted at 2026-09-29 03:31:15 on dist-test-slave-z2zg"
server_key: "4f54fc5c26e639741ecdaae6605c13e9"
server_key_iv: "6bcf785198aa1d718d24bbfdc2041159"
server_key_version: "encryptionkey@0"
I20260929 03:31:15.382110 28604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:15.389523 28604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:15.391728 28604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:15.391880 28604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:15.395695 28604 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38135
I20260929 03:31:15.395725 28671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38135 every 8 connection(s)
I20260929 03:31:15.396041 28604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:15.396795 28672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:15.398792 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265: Bootstrap starting.
I20260929 03:31:15.399358 28672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:15.399590 28672 log.cc:824] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:15.400179 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265: No bootstrap required, opened a new log
I20260929 03:31:15.401618 28672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } }
I20260929 03:31:15.401759 28672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:15.401780 28672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b7be69a15e41f4a8e2add292479265, State: Initialized, Role: FOLLOWER
I20260929 03:31:15.401814 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28604
I20260929 03:31:15.401948 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:15.401923 28672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } }
I20260929 03:31:15.402002 28672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:15.402055 28672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:15.402101 28672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:15.402158 6610 external_mini_cluster.cc:1428] Setting key 657ed6760ccc135e34e780cc4a7639c3
I20260929 03:31:15.403294 28672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } }
I20260929 03:31:15.403402 28672 leader_election.cc:302] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b7be69a15e41f4a8e2add292479265; no voters:
I20260929 03:31:15.403558 28672 leader_election.cc:288] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:15.403625 28677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:15.403781 28672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:15.403789 28677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [term 1 LEADER]: Becoming Leader. State: Replica: 31b7be69a15e41f4a8e2add292479265, State: Running, Role: LEADER
I20260929 03:31:15.404004 28677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } }
I20260929 03:31:15.404632 28679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b7be69a15e41f4a8e2add292479265. Latest consensus state: current_term: 1 leader_uuid: "31b7be69a15e41f4a8e2add292479265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } } }
I20260929 03:31:15.404748 28679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:15.404860 28678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b7be69a15e41f4a8e2add292479265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b7be69a15e41f4a8e2add292479265" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38135 } } }
I20260929 03:31:15.405019 28678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:15.405208 28686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:15.405709 28686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:15.407388 28686 catalog_manager.cc:1347] Generated new cluster ID: 0c54a46f1e3e4e0baa427cbbab4aab84
I20260929 03:31:15.407459 28686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:15.408306 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:15.402611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45543 (local address 127.6.116.190:38135)
0929 03:31:15.402837 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:15.402848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:15.402885 (+ 37us) server_negotiation.cc:407] Connection header received
0929 03:31:15.403024 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:15.403032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:15.403113 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:15.403252 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:15.403447 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:15.404902 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:15.406036 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:15.406602 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:15.406721 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:15.407534 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:15.407538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:15.407539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:15.407749 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:15.407784 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:15.407799 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:15.407868 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:15.408043 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 03:31:15.408123 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":116,"threads_started":1}
I20260929 03:31:15.409128 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:15.413894 28676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:15.410526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50597 (local address 127.6.116.190:38135)
0929 03:31:15.410559 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:15.410560 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:15.410751 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:31:15.410808 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:15.410809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:15.410817 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:15.410866 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:15.410983 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:15.411690 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:15.412721 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:15.413017 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:15.413079 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:15.413670 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:15.413671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:15.413671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:15.413676 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:15.413689 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:15.413694 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:15.413745 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:15.413808 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 03:31:15.413829 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:15.417341 28686 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:15.418013 28686 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:15.424983 28686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b7be69a15e41f4a8e2add292479265: Generated new TSK 0
I20260929 03:31:15.425171 28686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:15.428715 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28604
2026-09-29T03:31:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:15 dist-test-slave-z2zg krb5kdc[28703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:15 dist-test-slave-z2zg krb5kdc[28703](info): set up 2 sockets
Sep 29 03:31:15 dist-test-slave-z2zg krb5kdc[28703](info): commencing operation
krb5kdc: starting...
W20260929 03:31:16.452988 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:16 dist-test-slave-z2zg krb5kdc[28703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:16.565812 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38695
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:16.645828 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:16.646059 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:16.646095 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:16.647215 28716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:16.647284 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:16.647310 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:16.647342 28716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:16.647372 28716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:16.648761 28716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:16.649012 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:16.649240 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:16.653776 28724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:16.653793 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:16.653834 28721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:16.654052 28716 server_base.cc:1034] running on GCE node
I20260929 03:31:16.654309 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:16.654568 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:16.655725 28716 hybrid_clock.cc:648] HybridClock initialized: now 1790652676655719 us; error 36 us; skew 500 ppm
Sep 29 03:31:16 dist-test-slave-z2zg krb5kdc[28703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:16.662256 28716 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:16.662779 28716 webserver.cc:460] Webserver started at http://127.6.116.190:46511/ using document root <none> and password file <none>
I20260929 03:31:16.662997 28716 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:16.663079 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:16.663213 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:16.664177 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "ea47828b78654a28bbc5ab28522d1d6c"
format_stamp: "Formatted at 2026-09-29 03:31:16 on dist-test-slave-z2zg"
server_key: "807493ae32792a3727134d46ffb08c44"
server_key_iv: "f497259810f18b90e61bb1e0c623b18a"
server_key_version: "encryptionkey@0"
I20260929 03:31:16.664516 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "ea47828b78654a28bbc5ab28522d1d6c"
format_stamp: "Formatted at 2026-09-29 03:31:16 on dist-test-slave-z2zg"
server_key: "807493ae32792a3727134d46ffb08c44"
server_key_iv: "f497259810f18b90e61bb1e0c623b18a"
server_key_version: "encryptionkey@0"
I20260929 03:31:16.665817 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 03:31:16.666563 28731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:16.666728 28716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:16.666824 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "ea47828b78654a28bbc5ab28522d1d6c"
format_stamp: "Formatted at 2026-09-29 03:31:16 on dist-test-slave-z2zg"
server_key: "807493ae32792a3727134d46ffb08c44"
server_key_iv: "f497259810f18b90e61bb1e0c623b18a"
server_key_version: "encryptionkey@0"
I20260929 03:31:16.666903 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:16.670002 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:16.671861 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:16.672024 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:16.675897 28716 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38695
I20260929 03:31:16.675940 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38695 every 8 connection(s)
I20260929 03:31:16.676257 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:16.677134 28784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:16.679188 28784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c: Bootstrap starting.
I20260929 03:31:16.679760 28784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:16.679827 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28716
I20260929 03:31:16.679956 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:16.680105 28784 log.cc:824] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:16.680166 6610 external_mini_cluster.cc:1428] Setting key aa5eb9841853001d0d39676cd59aa66e
I20260929 03:31:16.681066 28784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c: No bootstrap required, opened a new log
I20260929 03:31:16.682622 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:16.680596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57637 (local address 127.6.116.190:38695)
0929 03:31:16.680758 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:16.680767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:16.680797 (+ 30us) server_negotiation.cc:407] Connection header received
0929 03:31:16.681381 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:16.681390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:16.681421 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:16.681567 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:16.682218 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:16.682221 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:16.682222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:16.682226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:16.682271 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:16.682284 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:16.682373 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:16.682405 (+ 32us) server_negotiation.cc:299] Negotiation successful
0929 03:31:16.682440 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260929 03:31:16.683691 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:16.683632 28784 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } }
I20260929 03:31:16.683779 28784 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 03:31:16.683820 28784 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea47828b78654a28bbc5ab28522d1d6c, State: Initialized, Role: FOLLOWER
I20260929 03:31:16.683924 28784 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } }
I20260929 03:31:16.683998 28784 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 03:31:16.684026 28784 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 03:31:16.684067 28784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:16.684752 28784 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } }
I20260929 03:31:16.684875 28784 leader_election.cc:302] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea47828b78654a28bbc5ab28522d1d6c; no voters:
I20260929 03:31:16.685032 28784 leader_election.cc:288] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 03:31:16.685086 28793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:16.685240 28793 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [term 1 LEADER]: Becoming Leader. State: Replica: ea47828b78654a28bbc5ab28522d1d6c, State: Running, Role: LEADER
I20260929 03:31:16.685264 28784 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:16.685412 28793 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } }
I20260929 03:31:16.685715 28793 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea47828b78654a28bbc5ab28522d1d6c. Latest consensus state: current_term: 1 leader_uuid: "ea47828b78654a28bbc5ab28522d1d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } } }
I20260929 03:31:16.685742 28794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea47828b78654a28bbc5ab28522d1d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea47828b78654a28bbc5ab28522d1d6c" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38695 } } }
I20260929 03:31:16.685883 28794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:16.685849 28793 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:16.686134 28803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:16.686642 28803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:16.688118 28803 catalog_manager.cc:1347] Generated new cluster ID: 8af06a0449b84016a2cfaf13a57c2a75
I20260929 03:31:16.688170 28803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:16.688282 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:16.685324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33527 (local address 127.6.116.190:38695)
0929 03:31:16.685374 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:16.685376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:16.686635 (+ 1259us) server_negotiation.cc:407] Connection header received
0929 03:31:16.686857 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:16.686858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:16.686863 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:16.686910 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:16.687932 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:16.687934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:16.687934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 03:31:16.687938 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:16.687950 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 03:31:16.687953 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:16.688004 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:16.688152 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 03:31:16.688165 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:16.701325 28803 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:16.701464 28803 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:16.706645 28803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea47828b78654a28bbc5ab28522d1d6c: Generated new TSK 0
I20260929 03:31:16.706794 28803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:16.716665 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28716
2026-09-29T03:31:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1287 ms)
[----------] 18 tests from AuthTokenIssuingTest (23140 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:16 dist-test-slave-z2zg krb5kdc[28815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:16 dist-test-slave-z2zg krb5kdc[28815](info): set up 2 sockets
Sep 29 03:31:16 dist-test-slave-z2zg krb5kdc[28815](info): commencing operation
krb5kdc: starting...
W20260929 03:31:17.741108 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:17 dist-test-slave-z2zg krb5kdc[28815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:17.851697 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38817
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:17.928915 28828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:17.929111 28828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:17.929127 28828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:17.929147 28828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:17.930250 28828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:17.930286 28828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:17.930300 28828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:17.930312 28828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:17.930359 28828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:17.931735 28828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:38817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:17.931964 28828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:17.932178 28828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:17.936841 28833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:17.936892 28834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:17.936851 28836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:17.936993 28828 server_base.cc:1034] running on GCE node
I20260929 03:31:17.937290 28828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:17.937572 28828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:17.938709 28828 hybrid_clock.cc:648] HybridClock initialized: now 1790652677938697 us; error 28 us; skew 500 ppm
Sep 29 03:31:17 dist-test-slave-z2zg krb5kdc[28815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:17.941036 28828 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:17.941466 28828 webserver.cc:460] Webserver started at http://127.6.116.190:37467/ using document root <none> and password file <none>
I20260929 03:31:17.941679 28828 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:17.941722 28828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:17.941869 28828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:17.942826 28828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "04c3af87acf8491c96c505757e5110f9"
format_stamp: "Formatted at 2026-09-29 03:31:17 on dist-test-slave-z2zg"
server_key: "ca216490178358f55fbc016a569cc8aa"
server_key_iv: "7c12fc639d816c07f072388519926400"
server_key_version: "encryptionkey@0"
I20260929 03:31:17.943193 28828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "04c3af87acf8491c96c505757e5110f9"
format_stamp: "Formatted at 2026-09-29 03:31:17 on dist-test-slave-z2zg"
server_key: "ca216490178358f55fbc016a569cc8aa"
server_key_iv: "7c12fc639d816c07f072388519926400"
server_key_version: "encryptionkey@0"
I20260929 03:31:17.944453 28828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:17.945132 28843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:17.945287 28828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 03:31:17.945371 28828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "04c3af87acf8491c96c505757e5110f9"
format_stamp: "Formatted at 2026-09-29 03:31:17 on dist-test-slave-z2zg"
server_key: "ca216490178358f55fbc016a569cc8aa"
server_key_iv: "7c12fc639d816c07f072388519926400"
server_key_version: "encryptionkey@0"
I20260929 03:31:17.945422 28828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:17.954927 28828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:17.956661 28828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:17.956763 28828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:17.960781 28828 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:38817
I20260929 03:31:17.960811 28895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:38817 every 8 connection(s)
I20260929 03:31:17.961138 28828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:17.962036 28896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:17.963590 28896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:17.965377 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28828
I20260929 03:31:17.965481 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:17.965690 6610 external_mini_cluster.cc:1428] Setting key e00b4eba3da972df75962b407cb6e280
Sep 29 03:31:17 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.116.189@KRBTEST.COM; defaulting to no policy
I20260929 03:31:17.973511 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:17.964231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.6.116.190:38817)
0929 03:31:17.964740 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:17.964744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:17.964779 (+ 35us) server_negotiation.cc:407] Connection header received
0929 03:31:17.965072 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:17.965074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:17.965088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:17.965160 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:17.965901 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:17.966712 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:17.967701 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:17.968009 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:17.968135 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:17.970359 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:17.970366 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:17.970367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:17.970373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:17.971945 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:17.972325 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:17.972327 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:17.972328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:17.972401 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:17.972620 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:17.972621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:17.972621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:17.972853 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:17.972956 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:17.973141 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 03:31:17.973291 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":69,"threads_started":1}
I20260929 03:31:17.973513 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:17.964202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38817 (local address 127.0.0.1:37734)
0929 03:31:17.964364 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:17.964408 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:17.964936 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:17.965185 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:17.965204 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:17.965545 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:17.965794 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:17.965800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:17.966829 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:17.966830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:17.967608 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:17.967610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:17.968120 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:17.968121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:17.968182 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:17.968706 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:17.968713 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:17.970259 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:17.972039 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:17.972045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:17.972046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:17.972233 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:17.972468 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:17.972469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:17.972470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:17.972554 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:17.972928 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:17.972934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:17.972991 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 03:31:17.973109 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 03:31:17.973351 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":118,"thread_start_us":84,"threads_started":1}
I20260929 03:31:17.974246 28896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:17.974557 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:17.974444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49164)
0929 03:31:17.974489 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:17.974528 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:17.974680 28847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
I20260929 03:31:17.975097 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:17.974977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49166)
0929 03:31:17.975024 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:17.975077 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:31:17.975198 28896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:46333: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
I20260929 03:31:17.989640 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:46333
--webserver_interface=127.6.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:31:18.024468 28896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } attempt: 1
I20260929 03:31:18.025017 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.024865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49172)
0929 03:31:18.024932 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.024980 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:31:18.025383 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.025296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49182)
0929 03:31:18.025338 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.025365 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:18.025487 28896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:46333: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
W20260929 03:31:18.068413 28903 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:18.068595 28903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:18.068612 28903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:18.068631 28903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:18.069720 28903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:18.069765 28903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:18.069778 28903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:18.069814 28903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:18.069844 28903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:18.071383 28903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:18.071565 28903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:18.071785 28903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:18.076272 28909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:18.076347 28908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:18.076459 28903 server_base.cc:1034] running on GCE node
W20260929 03:31:18.076480 28911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:18.076727 28903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:18.077016 28903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:18.078186 28903 hybrid_clock.cc:648] HybridClock initialized: now 1790652678078173 us; error 41 us; skew 500 ppm
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:18.080523 28903 init.cc:377] Logged in from keytab as kudu/127.6.116.189@KRBTEST.COM (short username kudu)
I20260929 03:31:18.080967 28903 webserver.cc:460] Webserver started at http://127.6.116.189:35681/ using document root <none> and password file <none>
I20260929 03:31:18.081184 28903 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:18.081251 28903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:18.081395 28903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:18.082355 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data/instance:
uuid: "0378048db3cc4f14a260fddb4c6fa27c"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "ffdea18e6ced326f9e2924bd04126205"
server_key_iv: "05f015d9923bd8bb9a0b40e6f23d9ddf"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.082688 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal/instance:
uuid: "0378048db3cc4f14a260fddb4c6fa27c"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "ffdea18e6ced326f9e2924bd04126205"
server_key_iv: "05f015d9923bd8bb9a0b40e6f23d9ddf"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.084028 28903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:31:18.084724 28918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:18.084880 28903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 03:31:18.085002 28903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "0378048db3cc4f14a260fddb4c6fa27c"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "ffdea18e6ced326f9e2924bd04126205"
server_key_iv: "05f015d9923bd8bb9a0b40e6f23d9ddf"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.085078 28903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:18.093616 28896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } attempt: 2
I20260929 03:31:18.093943 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.093829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49188)
0929 03:31:18.093883 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.093916 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 03:31:18.094319 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.094240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49192)
0929 03:31:18.094280 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.094304 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 03:31:18.094434 28896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:46333: Network error: Client connection negotiation failed: client connection to 127.6.116.189:46333: connect: Connection refused (error 111)
I20260929 03:31:18.106513 28903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:18.108299 28903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:18.108450 28903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:18.112462 28903 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.189:46333
I20260929 03:31:18.112501 28970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.189:46333 every 8 connection(s)
I20260929 03:31:18.112830 28903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
I20260929 03:31:18.113713 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28903
I20260929 03:31:18.113770 28971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:18.113823 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-1/wal/instance
I20260929 03:31:18.114007 6610 external_mini_cluster.cc:1428] Setting key d5f48ba446c71845b4030e972e38482f
I20260929 03:31:18.115317 28971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.116.188@KRBTEST.COM; defaulting to no policy
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.188@KRBTEST.COM" created.
I20260929 03:31:18.124234 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.115978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.6.116.190:38817)
0929 03:31:18.116043 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.116046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.116261 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:31:18.116737 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.116738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.116750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.116814 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.117483 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.117971 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.118922 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.119207 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.119292 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.121536 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.121537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.121537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.121544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.122896 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.123326 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.123327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.123327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.123379 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.123599 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.123600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.123600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.123710 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.123789 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.124069 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.124148 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:31:18.124377 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.115987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38817 (local address 127.0.0.1:37746)
0929 03:31:18.116131 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.116168 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.116632 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.116878 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.116881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.117206 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.117382 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.117386 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.118063 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.118064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.118829 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.118831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.119319 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.119320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.119449 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.119935 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.119943 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.121456 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.122979 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.122986 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.122988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.123261 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.123438 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.123439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.123439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.123528 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.123782 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.123789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.123919 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.124019 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.124165 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260929 03:31:18.125293 28971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:18.132196 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.125533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49196)
0929 03:31:18.125590 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.125600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.125643 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.125887 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.125887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.126014 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.126049 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.126051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.126875 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.126876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.127464 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.127465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.127967 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.127968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.128031 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.128424 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.128429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.129501 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.131312 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.131313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.131313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.131459 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.131641 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.131642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.131642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.131692 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.131880 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.131880 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.131913 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.131986 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.132106 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 03:31:18.132302 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.125583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.6.116.189:46333)
0929 03:31:18.125740 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.125744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.125763 (+ 19us) server_negotiation.cc:407] Connection header received
0929 03:31:18.125800 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.125801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.125810 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.125886 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.126123 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.126759 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.127551 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.127864 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.127967 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.129584 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.129591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.129592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.129599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.131247 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.131531 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.131533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.131533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.131583 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.131730 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.131730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.131730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.131832 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.131883 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.132018 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.132179 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260929 03:31:18.132905 28971 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:18.133270 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.133108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57482)
0929 03:31:18.133141 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.133238 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260929 03:31:18.133533 28921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
I20260929 03:31:18.134035 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.133931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57484)
0929 03:31:18.133957 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.133981 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 03:31:18.134157 28971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:37195: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
I20260929 03:31:18.138477 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:37195
--webserver_interface=127.6.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:31:18.183383 28971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } attempt: 1
I20260929 03:31:18.183851 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.183710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57490)
0929 03:31:18.183773 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.183816 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 03:31:18.184228 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.184140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57504)
0929 03:31:18.184183 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.184210 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:18.184348 28971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:37195: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
I20260929 03:31:18.185595 28896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } attempt: 3
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:18.192356 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.185866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49198)
0929 03:31:18.185922 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.185932 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.185983 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.186150 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.186151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.186301 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.186360 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.186362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.187005 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.187006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.187729 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.187731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.188139 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.188140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.188233 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.188728 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.188735 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.189827 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.191507 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.191507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.191508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.191661 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.191858 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.191859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.191859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.191908 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.192150 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.192151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.192186 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.192233 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.192299 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:31:18.192411 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.185928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.6.116.189:46333)
0929 03:31:18.185967 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.185969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.185980 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:31:18.186053 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.186054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.186065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.186129 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.186437 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.186922 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.187810 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.188043 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.188138 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.189910 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.189918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.189918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.189925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.191422 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.191734 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.191735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.191735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.191795 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.191965 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.191966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.191967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.192090 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.192155 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.192254 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.192327 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:31:18.192781 28896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:18.193089 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.192990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57512)
0929 03:31:18.193033 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.193071 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 03:31:18.193778 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.193702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57514)
0929 03:31:18.193742 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.193764 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:18.193895 28896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:37195: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
W20260929 03:31:18.218591 28978 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:18.218771 28978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:18.218788 28978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:18.218807 28978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:18.219990 28978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:18.220029 28978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:18.220044 28978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:18.220057 28978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:18.220068 28978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:18.221516 28978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:38817,127.6.116.189:46333,127.6.116.188:37195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:37195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:18.221751 28978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:18.222011 28978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 03:31:18.225077 28896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } attempt: 1
I20260929 03:31:18.225494 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.225368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57528)
0929 03:31:18.225426 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.225461 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:31:18.226055 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.225947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57534)
0929 03:31:18.226000 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.226026 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 03:31:18.226174 28896 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:37195: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
W20260929 03:31:18.226449 28983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:18.226478 28984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:18.226513 28986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:18.227121 28978 server_base.cc:1034] running on GCE node
I20260929 03:31:18.227382 28978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:18.227649 28978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:18.228792 28978 hybrid_clock.cc:648] HybridClock initialized: now 1790652678228769 us; error 32 us; skew 500 ppm
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:18.231199 28978 init.cc:377] Logged in from keytab as kudu/127.6.116.188@KRBTEST.COM (short username kudu)
I20260929 03:31:18.231606 28978 webserver.cc:460] Webserver started at http://127.6.116.188:37067/ using document root <none> and password file <none>
I20260929 03:31:18.231827 28978 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:18.231894 28978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:18.232054 28978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:18.233058 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data/instance:
uuid: "05a1b12a42844c06bdbad8a359438ac4"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "4e64f4448ac8cae22ed7bf571b5d0979"
server_key_iv: "3dd995f70ccbd0beb6bb36474aecf144"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.233408 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal/instance:
uuid: "05a1b12a42844c06bdbad8a359438ac4"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "4e64f4448ac8cae22ed7bf571b5d0979"
server_key_iv: "3dd995f70ccbd0beb6bb36474aecf144"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.234766 28978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 03:31:18.235502 28993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:18.235671 28978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:18.235766 28978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "05a1b12a42844c06bdbad8a359438ac4"
format_stamp: "Formatted at 2026-09-29 03:31:18 on dist-test-slave-z2zg"
server_key: "4e64f4448ac8cae22ed7bf571b5d0979"
server_key_iv: "3dd995f70ccbd0beb6bb36474aecf144"
server_key_version: "encryptionkey@0"
I20260929 03:31:18.235850 28978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:18.252161 28978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:18.252502 28971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } attempt: 2
I20260929 03:31:18.252856 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.252732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57538)
0929 03:31:18.252791 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.252821 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:31:18.253159 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.253068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57552)
0929 03:31:18.253112 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.253136 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:18.253291 28971 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:37195: Network error: Client connection negotiation failed: client connection to 127.6.116.188:37195: connect: Connection refused (error 111)
I20260929 03:31:18.254272 28978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:18.254424 28978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:18.258153 28978 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.188:37195
I20260929 03:31:18.258214 29045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.188:37195 every 8 connection(s)
I20260929 03:31:18.258528 28978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
I20260929 03:31:18.259480 29046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:18.260902 29046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:18.262584 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 28978
I20260929 03:31:18.262657 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790652420563207-6610-0/minicluster-data/master-2/wal/instance
I20260929 03:31:18.262820 6610 external_mini_cluster.cc:1428] Setting key 644ede6ea0e2e0c804fd957d31772353
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:18.269796 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.261517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.6.116.190:38817)
0929 03:31:18.261587 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.261588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.261785 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:31:18.262214 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.262215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.262223 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.262274 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.262920 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.263674 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.264586 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.264796 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.264885 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.266813 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.266814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.266815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.266823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.268279 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.268726 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.268728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.268728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.268785 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.269103 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.269104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.269104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.269249 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.269323 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.269606 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.269685 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:31:18.269932 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.261502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:38817 (local address 127.0.0.1:37750)
0929 03:31:18.261666 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.261703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.262138 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.262324 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.262326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.262580 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.262811 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.262816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.263744 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.263745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.264487 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.264489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.264913 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.264914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.265030 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.265450 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.265457 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.266738 (+ 1281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.268358 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.268364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.268366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.268660 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.268870 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.268871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.268872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.269003 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.269320 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.269325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.269452 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.269563 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.269741 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260929 03:31:18.270558 29046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:18.271855 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.263264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.6.116.188:37195)
0929 03:31:18.263425 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.263429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.263467 (+ 38us) server_negotiation.cc:407] Connection header received
0929 03:31:18.263528 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.263530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.263546 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.263605 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.264241 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.264922 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.265963 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.266258 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.266349 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.268604 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.268610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.268610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.268615 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.270510 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.271134 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.271137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.271137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.271193 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.271411 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.271412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.271412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.271528 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.271627 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.271740 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.271793 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:18.277269 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.270802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:46333 (local address 127.0.0.1:49212)
0929 03:31:18.270892 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.270901 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.270933 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.271091 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.271092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.271238 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.271272 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.271274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.272071 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.272071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.272680 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.272682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.273207 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.273208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.273273 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.273590 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.273595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.274700 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.276300 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.276301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.276301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.276499 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.276676 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.276677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.276677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.276736 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.276975 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.276976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.277019 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.277105 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.277186 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 03:31:18.277299 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.270856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.6.116.189:46333)
0929 03:31:18.270914 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.270916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.270926 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:31:18.270977 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.270978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.270989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.271098 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.271341 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.272001 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.272760 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.273126 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.273196 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.274778 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.274779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.274779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.274785 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.276228 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.276573 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.276574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.276574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.276618 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.276800 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.276801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.276801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.276908 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.276955 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.277120 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.277198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 03:31:18.277613 29046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:18.280257 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.272631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.6.116.190:38817)
0929 03:31:18.272667 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.272669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.272678 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:31:18.272698 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.272698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.272707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.272761 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.273315 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.274011 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.274726 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.274953 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.275027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.277202 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.277203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.277204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.277210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.279209 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.279545 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.279545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.279546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.279590 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.279786 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.279786 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.279787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.279870 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.280037 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.280102 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.280170 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:18.284503 29048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.277857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57562)
0929 03:31:18.278030 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.278045 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.278080 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.278197 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.278198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.278294 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.278316 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.278317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.279168 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.279169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.279889 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.279891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.280383 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.280384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.280433 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.280798 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.280802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.281993 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.283639 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.283639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.283640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.283794 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.283993 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.283994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.283994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.284041 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.284304 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.284305 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.284338 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.284391 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.284451 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260929 03:31:18.284629 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.277902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.6.116.188:37195)
0929 03:31:18.277957 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.277958 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.278088 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:31:18.278123 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.278124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.278135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.278196 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.278404 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.279096 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.280012 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.280321 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.280386 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.282100 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.282101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.282101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.282107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.283555 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.283867 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.283868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.283868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.283921 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.284099 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.284100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.284100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.284231 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.284302 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.284436 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.284504 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:18.285940 29046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: Bootstrap starting.
I20260929 03:31:18.286979 29046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:18.287322 29046 log.cc:824] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:18.288184 29046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: No bootstrap required, opened a new log
I20260929 03:31:18.288412 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.280867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.6.116.189:46333)
0929 03:31:18.280906 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.280909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.280917 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:18.280937 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.280938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.280947 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.280996 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.281517 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.281978 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.282934 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.283186 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.283238 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.285686 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.285687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.285688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.285695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.287296 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.287699 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.287700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.287700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.287749 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.287940 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.287941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.287941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.288068 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.288215 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.288255 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.288332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:18.289732 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:18.289907 29046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.290025 29046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a1b12a42844c06bdbad8a359438ac4, State: Initialized, Role: FOLLOWER
I20260929 03:31:18.290114 29046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.290395 29046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:18.290351 29052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.290457 29052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.290751 28870 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "04c3af87acf8491c96c505757e5110f9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37760
W20260929 03:31:18.292155 29063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:18.292207 29063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:18.292393 28978 master_runner.cc:186] Error getting master registration for 127.6.116.190:38817: OK, instance_id { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" instance_seqno: 1790652677959704 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:31:18.295138 28978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:31:18.301309 28896 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } attempt: 2
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652677, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:18.308604 28898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.301549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57564)
0929 03:31:18.301604 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.301616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.301661 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.301769 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.301769 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.301963 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.302005 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.302007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.302859 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.302859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.303717 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.303720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.304170 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.304171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.304232 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.304650 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.304655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.306047 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.307569 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.307570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.307570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.307734 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.307936 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.307937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.307937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.307991 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.308290 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.308292 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.308334 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.308418 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.308512 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 03:31:18.308604 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.301607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.6.116.188:37195)
0929 03:31:18.301641 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.301642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.301651 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:31:18.301710 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.301711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.301717 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.301755 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.302095 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.302775 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.303798 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.304094 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.304158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.306134 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.306135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.306135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.306142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.307466 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.307816 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.307817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.307817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.307871 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.308075 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.308076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.308076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.308214 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.308286 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.308443 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.308511 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:31:18.309749 28896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: Bootstrap starting.
I20260929 03:31:18.310323 28896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:18.310603 28896 log.cc:824] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:18.311308 28896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: No bootstrap required, opened a new log
I20260929 03:31:18.312255 28896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.312356 28896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c3af87acf8491c96c505757e5110f9, State: Initialized, Role: FOLLOWER
I20260929 03:31:18.312458 28896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.312738 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.312795 28896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:18.312813 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.312922 28870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:31:18.313002 28870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:18.313784 28870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.314272 29020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04c3af87acf8491c96c505757e5110f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "05a1b12a42844c06bdbad8a359438ac4"
I20260929 03:31:18.314389 29020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:18.315234 29020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c3af87acf8491c96c505757e5110f9 in term 1.
I20260929 03:31:18.315449 28846 leader_election.cc:302] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c3af87acf8491c96c505757e5110f9, 05a1b12a42844c06bdbad8a359438ac4; no voters:
I20260929 03:31:18.315568 29066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:18.315706 29066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [term 1 LEADER]: Becoming Leader. State: Replica: 04c3af87acf8491c96c505757e5110f9, State: Running, Role: LEADER
I20260929 03:31:18.315783 29066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.316186 28870 leader_election.cc:288] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [CANDIDATE]: Term 1 election: Requested vote from peers 0378048db3cc4f14a260fddb4c6fa27c (127.6.116.189:46333), 05a1b12a42844c06bdbad8a359438ac4 (127.6.116.188:37195)
I20260929 03:31:18.316149 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c3af87acf8491c96c505757e5110f9. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.316251 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: This master's current role is: LEADER
W20260929 03:31:18.316125 29077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:18.316540 29077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:18.315275 28828 master_runner.cc:186] Error getting master registration for 127.6.116.189:46333: OK, instance_id { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" instance_seqno: 1790652678111362 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:31:18.317701 28828 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:31:18.319160 29081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:18.319334 29081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:18.321627 29020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [term 1 FOLLOWER]: Refusing update from remote peer 04c3af87acf8491c96c505757e5110f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:31:18.321913 29066 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:31:18.323040 29052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c3af87acf8491c96c505757e5110f9. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.323167 29052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.325628 29052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.325731 29052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.326175 29066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.326254 29066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:18.326365 29089 mvcc.cc:205] Tried to move back new op lower bound from 7334513370403262464 to 7334513370382082048. Current Snapshot: MvccSnapshot[applied={T|T < 7334513370403262464}]
I20260929 03:31:18.327119 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.317242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37768 (local address 127.6.116.190:38817)
0929 03:31:18.317277 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.317279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.317884 (+ 605us) server_negotiation.cc:407] Connection header received
0929 03:31:18.317942 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.317943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.317967 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.318023 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.318704 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.319316 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.320466 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.320660 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.320734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.322215 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.322217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.322217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.322224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.324020 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.326213 (+ 2193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.326214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.326214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.326267 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.326472 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.326473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.326473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.326614 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.326704 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.326920 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.327000 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:31:18.328424 29081 catalog_manager.cc:1347] Generated new cluster ID: 197d5dc880c947eab1e6e75c750e191c
I20260929 03:31:18.328474 29081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:18.328758 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.317234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.6.116.189:46333)
0929 03:31:18.317271 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.317273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.318234 (+ 961us) server_negotiation.cc:407] Connection header received
0929 03:31:18.318336 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.318337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.318347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.318375 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.319075 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.319734 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.320871 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.321661 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.321739 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.324362 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.324364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.324364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.324371 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.326032 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.326845 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.326846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.326846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.326934 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.327586 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.327587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.327588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.327806 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.327927 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.328319 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.328454 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:18.330020 29090 mvcc.cc:205] Tried to move back new op lower bound from 7334513370403262464 to 7334513370382082048. Current Snapshot: MvccSnapshot[applied={T|T < 7334513370403262464}]
I20260929 03:31:18.332093 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.317567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.6.116.188:37195)
0929 03:31:18.317609 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.317611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.323672 (+ 6061us) server_negotiation.cc:407] Connection header received
0929 03:31:18.323808 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.323809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.323818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.323845 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.324810 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.325567 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.327375 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.327661 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.327698 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.329101 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.329103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.329103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.329110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.330810 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.331235 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.331236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.331236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.331290 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.331507 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.331508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.331508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.331631 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.331723 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.331902 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.331975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 03:31:18.337623 29081 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:18.338327 29081 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:18.344400 28971 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } attempt: 3
Sep 29 03:31:18 dist-test-slave-z2zg krb5kdc[28815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652678, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:18.349159 29081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9: Generated new TSK 0
I20260929 03:31:18.349326 29081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 03:31:18.352257 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.344633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:37195 (local address 127.0.0.1:57576)
0929 03:31:18.344678 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:18.344689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:18.344726 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:18.344919 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:18.344920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:18.345088 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:18.345134 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.345136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.345988 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.345989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.346774 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:18.346776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.347272 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.347273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:18.347334 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.347824 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:18.347830 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:18.349374 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:18.351256 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.351257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.351258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.351460 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.351650 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:18.351650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:18.351651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:18.351703 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:18.351955 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:18.351956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:18.351997 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:31:18.352067 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:31:18.352157 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:31:18.352586 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.344692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.6.116.188:37195)
0929 03:31:18.344821 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.344823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.344829 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:31:18.344849 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.344850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.344858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.344905 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.345213 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.345909 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.346849 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.347164 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.347221 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.349482 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:18.349484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:18.349484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:18.349490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:18.351153 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.351532 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.351533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.351534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.351585 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:18.351763 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:18.351764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:18.351764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:18.351877 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:18.351932 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.352434 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.352503 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 03:31:18.353999 28971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: Bootstrap starting.
I20260929 03:31:18.354727 28971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:18.355010 28971 log.cc:824] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:18.355989 28971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: No bootstrap required, opened a new log
I20260929 03:31:18.357969 28971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.358090 28971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0378048db3cc4f14a260fddb4c6fa27c, State: Initialized, Role: FOLLOWER
I20260929 03:31:18.358161 28971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } }
I20260929 03:31:18.358449 28971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:18.358479 28945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04c3af87acf8491c96c505757e5110f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0378048db3cc4f14a260fddb4c6fa27c"
I20260929 03:31:18.358678 28945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [term 0 FOLLOWER]: Advancing to term 1
W20260929 03:31:18.359148 28924 tablet.cc:2369] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 03:31:18.359860 28945 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04c3af87acf8491c96c505757e5110f9 in term 1.
I20260929 03:31:18.360005 28944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [term 1 FOLLOWER]: Refusing update from remote peer 04c3af87acf8491c96c505757e5110f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:31:18.358435 29099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.360260 29099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.360272 29092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:31:18.362015 29111 mvcc.cc:205] Tried to move back new op lower bound from 7334513370512220160 to 7334513370382082048. Current Snapshot: MvccSnapshot[applied={T|T < 7334513370403262464}]
I20260929 03:31:18.362382 29099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04c3af87acf8491c96c505757e5110f9. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.362493 29099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.362556 29099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.362629 29099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:18.364351 29110 catalog_manager.cc:1259] Loaded cluster ID: 197d5dc880c947eab1e6e75c750e191c
I20260929 03:31:18.364427 29110 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: loading cluster ID for follower catalog manager: success
I20260929 03:31:18.365582 29110 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: acquiring CA information for follower catalog manager: success
I20260929 03:31:18.365932 29110 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0378048db3cc4f14a260fddb4c6fa27c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 03:31:18.380112 28899 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.378310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37774 (local address 127.6.116.190:38817)
0929 03:31:18.378349 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.378352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.379174 (+ 822us) server_negotiation.cc:407] Connection header received
0929 03:31:18.379492 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.379493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.379503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.379562 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.380052 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37774: BlockingRecv error: recv got EOF from 127.0.0.1:37774 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:31:18.380647 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.378518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.6.116.188:37195)
0929 03:31:18.378562 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.378564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.379258 (+ 694us) server_negotiation.cc:407] Connection header received
0929 03:31:18.379625 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.379626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.379635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.379662 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.380594 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57578: BlockingRecv error: recv got EOF from 127.0.0.1:57578 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260929 03:31:18.381276 28976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.378711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.6.116.189:46333)
0929 03:31:18.378744 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.378746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.379404 (+ 658us) server_negotiation.cc:407] Connection header received
0929 03:31:18.380887 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.380888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.380897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.380925 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.381227 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49244: BlockingRecv error: recv got EOF from 127.0.0.1:49244 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:18.386242 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.382901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.6.116.189:46333)
0929 03:31:18.382939 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.382941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.383071 (+ 130us) server_negotiation.cc:407] Connection header received
0929 03:31:18.383176 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.383182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.383210 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.383252 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.383432 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.384198 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.385305 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.385613 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.385669 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.385879 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.386000 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.386048 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.386160 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.386179 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:31:18.389284 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.388614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.6.116.188:37195)
0929 03:31:18.388649 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.388650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.388824 (+ 174us) server_negotiation.cc:407] Connection header received
0929 03:31:18.388921 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.388921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.388936 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.388964 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.389240 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57594: BlockingRecv error: recv got EOF from 127.0.0.1:57594 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:18.394616 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.391515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.6.116.189:46333)
0929 03:31:18.391558 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.391560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.391766 (+ 206us) server_negotiation.cc:407] Connection header received
0929 03:31:18.391854 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.391855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.391863 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.391888 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.392039 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.392763 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.393839 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.394127 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.394165 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.394346 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.394376 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.394402 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.394527 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.394556 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 03:31:18.397364 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.396700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.6.116.188:37195)
0929 03:31:18.396732 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.396734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.396921 (+ 187us) server_negotiation.cc:407] Connection header received
0929 03:31:18.397008 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.397009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.397018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.397045 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.397323 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57600: BlockingRecv error: recv got EOF from 127.0.0.1:57600 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 03:31:18.403389 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.400319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.6.116.189:46333)
0929 03:31:18.400361 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.400363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.400580 (+ 217us) server_negotiation.cc:407] Connection header received
0929 03:31:18.400670 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.400671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.400678 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.400704 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.400851 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.401554 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.402620 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.402904 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.402940 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.403145 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.403175 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.403207 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.403310 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.403326 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:31:18.406085 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.405401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.6.116.188:37195)
0929 03:31:18.405432 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.405434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.405629 (+ 195us) server_negotiation.cc:407] Connection header received
0929 03:31:18.405737 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.405738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.405747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.405775 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.406045 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57612: BlockingRecv error: recv got EOF from 127.0.0.1:57612 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 03:31:18.414100 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.411034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.6.116.189:46333)
0929 03:31:18.411082 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.411084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.411299 (+ 215us) server_negotiation.cc:407] Connection header received
0929 03:31:18.411387 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.411388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.411396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.411422 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.411587 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.412280 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.413350 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.413634 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.413673 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.413862 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.413891 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.413917 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.414020 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.414039 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 03:31:18.416852 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.416158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.6.116.188:37195)
0929 03:31:18.416201 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.416203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.416400 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:31:18.416490 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.416491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.416499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.416526 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.416811 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57622: BlockingRecv error: recv got EOF from 127.0.0.1:57622 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 03:31:18.428907 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.425875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.6.116.189:46333)
0929 03:31:18.425908 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.425910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.426118 (+ 208us) server_negotiation.cc:407] Connection header received
0929 03:31:18.426207 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.426208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.426216 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.426242 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.426384 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.427121 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.428170 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.428428 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.428468 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.428673 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.428703 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.428730 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.428830 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.428845 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 03:31:18.431885 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.431130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.6.116.188:37195)
0929 03:31:18.431163 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.431165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.431393 (+ 228us) server_negotiation.cc:407] Connection header received
0929 03:31:18.431499 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.431500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.431509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.431547 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.431843 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57636: BlockingRecv error: recv got EOF from 127.0.0.1:57636 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 03:31:18.452325 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.449072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.6.116.189:46333)
0929 03:31:18.449107 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.449109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.449377 (+ 268us) server_negotiation.cc:407] Connection header received
0929 03:31:18.449474 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.449475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.449485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.449513 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.449708 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.450396 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.451529 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.451823 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.451870 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.452069 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.452101 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.452130 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.452238 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.452255 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:31:18.455360 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.454570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.6.116.188:37195)
0929 03:31:18.454605 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.454607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.454814 (+ 207us) server_negotiation.cc:407] Connection header received
0929 03:31:18.454908 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.454909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.454919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.454949 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.455313 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57648: BlockingRecv error: recv got EOF from 127.0.0.1:57648 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 03:31:18.491963 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.488688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.6.116.189:46333)
0929 03:31:18.488731 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.488733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.488999 (+ 266us) server_negotiation.cc:407] Connection header received
0929 03:31:18.489094 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.489095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.489104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.489131 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.489293 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.490004 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.491138 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.491440 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.491477 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.491712 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.491744 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.491770 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.491864 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.491882 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:31:18.495103 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.494261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.6.116.188:37195)
0929 03:31:18.494300 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.494302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.494493 (+ 191us) server_negotiation.cc:407] Connection header received
0929 03:31:18.494602 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.494603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.494613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.494640 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.495030 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57650: BlockingRecv error: recv got EOF from 127.0.0.1:57650 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:18.562661 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.560480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.6.116.189:46333)
0929 03:31:18.560560 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.560562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.560680 (+ 118us) server_negotiation.cc:407] Connection header received
0929 03:31:18.560738 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.560739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.560748 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.560790 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.560895 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.561359 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.562076 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.562248 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.562297 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.562421 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.562446 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.562483 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.562534 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.562554 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 03:31:18.564859 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.564318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.6.116.188:37195)
0929 03:31:18.564377 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.564379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.564415 (+ 36us) server_negotiation.cc:407] Connection header received
0929 03:31:18.564466 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.564467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.564477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.564522 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.564811 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57654: BlockingRecv error: recv got EOF from 127.0.0.1:57654 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 03:31:18.696197 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.693933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.6.116.189:46333)
0929 03:31:18.694005 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.694007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.694134 (+ 127us) server_negotiation.cc:407] Connection header received
0929 03:31:18.694190 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.694191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.694201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.694242 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.694361 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.694837 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.695569 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.695755 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.695812 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.695931 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.695957 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.696003 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.696062 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.696076 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 03:31:18.698356 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.697796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.6.116.188:37195)
0929 03:31:18.697859 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.697861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.697906 (+ 45us) server_negotiation.cc:407] Connection header received
0929 03:31:18.697950 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.697950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.697959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.698001 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.698284 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57664: BlockingRecv error: recv got EOF from 127.0.0.1:57664 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 03:31:18.957831 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:18.955453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.6.116.189:46333)
0929 03:31:18.955505 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.955516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.955542 (+ 26us) server_negotiation.cc:407] Connection header received
0929 03:31:18.955638 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.955639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.955649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.955714 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:18.955869 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.956353 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.957128 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:18.957320 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:18.957398 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:18.957562 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:18.957594 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:18.957652 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:18.957706 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 03:31:18.957743 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 03:31:18.960283 29050 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:18.959660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.6.116.188:37195)
0929 03:31:18.959724 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:18.959726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:18.959738 (+ 12us) server_negotiation.cc:407] Connection header received
0929 03:31:18.959808 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:18.959809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:18.959817 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:18.959888 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:18.960205 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57672: BlockingRecv error: recv got EOF from 127.0.0.1:57672 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:31:18.970415 29227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04c3af87acf8491c96c505757e5110f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04c3af87acf8491c96c505757e5110f9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 38817 } } peers { permanent_uuid: "0378048db3cc4f14a260fddb4c6fa27c" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 46333 } } peers { permanent_uuid: "05a1b12a42844c06bdbad8a359438ac4" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 37195 } } }
I20260929 03:31:18.970549 29227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c3af87acf8491c96c505757e5110f9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:19.292590 29063 catalog_manager.cc:1259] Loaded cluster ID: 197d5dc880c947eab1e6e75c750e191c
I20260929 03:31:19.292658 29063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: loading cluster ID for follower catalog manager: success
I20260929 03:31:19.293476 29063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: acquiring CA information for follower catalog manager: success
I20260929 03:31:19.293785 29063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05a1b12a42844c06bdbad8a359438ac4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:31:19.475875 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:19.473355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.6.116.189:46333)
0929 03:31:19.473478 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:19.473480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:19.473522 (+ 42us) server_negotiation.cc:407] Connection header received
0929 03:31:19.473604 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:19.473604 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:19.473615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:19.473696 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:19.473796 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.474385 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.475143 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.475350 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.475436 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:19.475596 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:19.475633 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:19.475693 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:19.475754 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 03:31:19.475775 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":81,"threads_started":1}
I20260929 03:31:19.479847 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:19.477368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.6.116.188:37195)
0929 03:31:19.477486 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:19.477488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:19.477496 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:19.477566 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:19.477567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:19.477642 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:19.477723 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:19.477826 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.478322 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.479070 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.479295 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.479379 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:19.479518 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:19.479633 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:19.479719 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:19.479741 (+ 22us) server_negotiation.cc:299] Negotiation successful
0929 03:31:19.479761 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260929 03:31:19.484061 29234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:19.481535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49336 (local address 127.6.116.189:46333)
0929 03:31:19.481612 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:19.481614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:19.481624 (+ 10us) server_negotiation.cc:407] Connection header received
0929 03:31:19.481650 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:19.481651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:19.481661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:19.481711 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:19.481822 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.482444 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.483300 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.483593 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.483659 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:19.483803 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:19.483832 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:19.483878 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:19.483959 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 03:31:19.483984 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 03:31:19.484854 29248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:19.481771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.6.116.190:38817)
0929 03:31:19.481885 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:19.481887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:19.481893 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:31:19.481954 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:19.481955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:19.482007 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:19.482083 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:19.482282 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.483033 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.483888 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.484175 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.484244 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:19.484466 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:19.484552 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:19.484627 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:19.484726 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 03:31:19.484752 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260929 03:31:19.485095 29240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:19.481889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.6.116.188:37195)
0929 03:31:19.481950 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:19.481952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:19.481959 (+ 7us) server_negotiation.cc:407] Connection header received
0929 03:31:19.481980 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:19.481981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:19.481990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:19.482057 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:19.482202 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.482898 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.484046 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:19.484535 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:19.484589 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:19.484746 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:19.484773 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:19.484825 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:19.484872 (+ 47us) server_negotiation.cc:299] Negotiation successful
0929 03:31:19.484890 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 03:31:19.486562 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28828
I20260929 03:31:19.492208 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28903
I20260929 03:31:19.496760 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 28978
2026-09-29T03:31:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2781 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 03:31:19 dist-test-slave-z2zg krb5kdc[29251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 03:31:19 dist-test-slave-z2zg krb5kdc[29251](info): set up 2 sockets
Sep 29 03:31:19 dist-test-slave-z2zg krb5kdc[29251](info): commencing operation
krb5kdc: starting...
W20260929 03:31:20.545176 6610 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/test-user.keytab.
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T03:31:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T03:31:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.116.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
Entry for principal kudu/127.6.116.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab.
I20260929 03:31:20.662153 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41005
--webserver_interface=127.6.116.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:20.740103 29264 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:20.740352 29264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:20.740383 29264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:20.740425 29264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:20.741523 29264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:20.741601 29264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:20.741626 29264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:20.741655 29264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:20.741686 29264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:20.743180 29264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.190.keytab
--principal=kudu/127.6.116.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.190:41005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.116.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:20.743460 29264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:20.743691 29264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:20.748355 29270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:20.748385 29272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:20.748386 29269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:20.748453 29264 server_base.cc:1034] running on GCE node
I20260929 03:31:20.748786 29264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:20.749068 29264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:20.750252 29264 hybrid_clock.cc:648] HybridClock initialized: now 1790652680750247 us; error 36 us; skew 500 ppm
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:20.752710 29264 init.cc:377] Logged in from keytab as kudu/127.6.116.190@KRBTEST.COM (short username kudu)
I20260929 03:31:20.753123 29264 webserver.cc:460] Webserver started at http://127.6.116.190:37519/ using document root <none> and password file <none>
I20260929 03:31:20.753360 29264 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:20.753429 29264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:20.753580 29264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:20.754570 29264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data/instance:
uuid: "07b582cdcd52440f8c0b13606d8442e9"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "8f11014254e3d117c3d58e97f0401daa"
server_key_iv: "437be50ce411eadb964196280dfd5947"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.754925 29264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance:
uuid: "07b582cdcd52440f8c0b13606d8442e9"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "8f11014254e3d117c3d58e97f0401daa"
server_key_iv: "437be50ce411eadb964196280dfd5947"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.756282 29264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:20.757005 29279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:20.757223 29264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:20.757329 29264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal
uuid: "07b582cdcd52440f8c0b13606d8442e9"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "8f11014254e3d117c3d58e97f0401daa"
server_key_iv: "437be50ce411eadb964196280dfd5947"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.757404 29264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:20.770432 29264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:20.772774 29264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:20.772935 29264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:20.776923 29264 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.190:41005
I20260929 03:31:20.776971 29331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.190:41005 every 8 connection(s)
I20260929 03:31:20.777343 29264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/data/info.pb
I20260929 03:31:20.778198 29332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:20.779856 29332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:20.786532 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 29264
I20260929 03:31:20.786667 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-0/wal/instance
I20260929 03:31:20.786854 6610 external_mini_cluster.cc:1428] Setting key a53b2b687ec9fb3de9ffa4bdda6a3780
I20260929 03:31:20.789117 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.780495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41005 (local address 127.0.0.1:45908)
0929 03:31:20.780662 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.780701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:20.781145 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:20.781382 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:20.781384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:20.781619 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:20.781768 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.781771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.782719 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.782720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.783463 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.783466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.783929 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.783930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.783981 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.784434 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:20.784442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:20.785729 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:20.787600 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.787604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.787606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.787831 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.788109 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.788110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.788111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.788197 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.788535 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:20.788540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:20.788609 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 03:31:20.788715 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 03:31:20.788940 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":96,"thread_start_us":70,"threads_started":1}
I20260929 03:31:20.789117 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.780548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.6.116.190:41005)
0929 03:31:20.780700 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:20.780707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:20.780798 (+ 91us) server_negotiation.cc:407] Connection header received
0929 03:31:20.781262 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:20.781264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:20.781279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:20.781350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:20.781845 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.782640 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.783533 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.783839 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.783917 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.785804 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:20.785811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:20.785811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:20.785817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:20.787500 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.787928 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.787931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.787931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.788013 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.788264 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.788264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.788265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.788463 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:20.788556 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:20.788750 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 03:31:20.788878 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":106,"threads_started":1}
I20260929 03:31:20.789950 29332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:20.790256 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.790150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:60968)
0929 03:31:20.790195 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.790234 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 03:31:20.790414 29280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
I20260929 03:31:20.790839 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.790754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:60974)
0929 03:31:20.790799 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.790823 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:31:20.790968 29332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:43285: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.116.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
Entry for principal kudu/127.6.116.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab.
I20260929 03:31:20.810448 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:43285
--webserver_interface=127.6.116.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 03:31:20.840250 29332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } attempt: 1
I20260929 03:31:20.840807 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.840655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:60984)
0929 03:31:20.840723 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.840772 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 03:31:20.841185 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.841097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:60988)
0929 03:31:20.841140 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.841166 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:20.841281 29332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:43285: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
W20260929 03:31:20.893090 29339 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:20.893307 29339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:20.893325 29339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:20.893347 29339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:20.894433 29339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:20.894467 29339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:20.894481 29339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:20.894495 29339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:20.894512 29339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:20.895921 29339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.189.keytab
--principal=kudu/127.6.116.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.189:43285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.116.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:20.896116 29339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:20.896287 29339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:20.901037 29347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:20.901108 29344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:20.901119 29345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:20.901286 29339 server_base.cc:1034] running on GCE node
I20260929 03:31:20.901739 29339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:20.902071 29339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:20.903249 29339 hybrid_clock.cc:648] HybridClock initialized: now 1790652680903240 us; error 35 us; skew 500 ppm
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:20.905814 29339 init.cc:377] Logged in from keytab as kudu/127.6.116.189@KRBTEST.COM (short username kudu)
I20260929 03:31:20.906231 29339 webserver.cc:460] Webserver started at http://127.6.116.189:41171/ using document root <none> and password file <none>
I20260929 03:31:20.906450 29339 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:20.906517 29339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:20.906669 29339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:20.907742 29339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data/instance:
uuid: "a08d6fbc2426422eb9d6faaf4d515287"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "96b42c89fc2085ad383e0344595004e4"
server_key_iv: "0055f461ae5307210818d14442d50fb2"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.908095 29339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal/instance:
uuid: "a08d6fbc2426422eb9d6faaf4d515287"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "96b42c89fc2085ad383e0344595004e4"
server_key_iv: "0055f461ae5307210818d14442d50fb2"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.909382 29339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 03:31:20.909406 29332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } attempt: 2
I20260929 03:31:20.909789 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.909671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:60992)
0929 03:31:20.909729 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.909764 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 03:31:20.910180 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.910074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:32772)
0929 03:31:20.910124 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.910158 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 03:31:20.910280 29332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.189:43285: Network error: Client connection negotiation failed: client connection to 127.6.116.189:43285: connect: Connection refused (error 111)
I20260929 03:31:20.910338 29354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:20.910514 29339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 03:31:20.910615 29339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal
uuid: "a08d6fbc2426422eb9d6faaf4d515287"
format_stamp: "Formatted at 2026-09-29 03:31:20 on dist-test-slave-z2zg"
server_key: "96b42c89fc2085ad383e0344595004e4"
server_key_iv: "0055f461ae5307210818d14442d50fb2"
server_key_version: "encryptionkey@0"
I20260929 03:31:20.910699 29339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:20.917673 29339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:20.919420 29339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:20.919574 29339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:20.923153 29339 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.189:43285
I20260929 03:31:20.923203 29406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.189:43285 every 8 connection(s)
I20260929 03:31:20.923553 29339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/data/info.pb
I20260929 03:31:20.924520 29407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:20.924741 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 29339
I20260929 03:31:20.924842 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-1/wal/instance
I20260929 03:31:20.925096 6610 external_mini_cluster.cc:1428] Setting key bc9e06a3d60aaf871214296e737a2ece
I20260929 03:31:20.927151 29407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.116.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.116.188@KRBTEST.COM" created.
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:20.938977 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.927830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.6.116.190:41005)
0929 03:31:20.927884 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:20.927886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:20.928067 (+ 181us) server_negotiation.cc:407] Connection header received
0929 03:31:20.928559 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:20.928560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:20.928573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:20.928635 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:20.929282 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.930087 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.931223 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.931459 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.931533 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.936048 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:20.936050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:20.936050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:20.936058 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:20.937556 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.937967 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.937968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.937968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.938031 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.938310 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.938311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.938312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.938459 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:20.938546 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:20.938785 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 03:31:20.938872 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:31:20.939170 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.927796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41005 (local address 127.0.0.1:45916)
0929 03:31:20.927954 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.927987 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:20.928456 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:20.928701 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:20.928706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:20.928959 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:20.929157 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.929162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.930187 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.930192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.931140 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.931142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.931568 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.931569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.931691 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.932191 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:20.932199 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:20.935954 (+ 3755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:20.937633 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.937638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.937640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.937875 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.938126 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.938127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.938127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.938233 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.938530 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:20.938534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:20.938639 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 03:31:20.938761 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 03:31:20.938912 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260929 03:31:20.940656 29407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:31:20 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:20.948565 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.940937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:32788)
0929 03:31:20.940994 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.941006 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:20.941054 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:20.941291 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:20.941292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:20.941447 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:20.941503 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.941504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.942345 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.942346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.943183 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:20.943185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.943687 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.943688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:20.943743 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.944121 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:20.944127 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:20.945546 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:20.947598 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.947599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.947599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.947748 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.947950 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:20.947951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:20.947951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:20.948042 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:20.948266 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:20.948267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:20.948300 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 03:31:20.948370 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:31:20.948454 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 03:31:20.948628 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.940979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32788 (local address 127.6.116.189:43285)
0929 03:31:20.941130 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:20.941134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:20.941155 (+ 21us) server_negotiation.cc:407] Connection header received
0929 03:31:20.941185 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:20.941187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:20.941199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:20.941269 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:20.941592 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.942252 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.943255 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:20.943612 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:20.943677 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:20.945672 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:20.945679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:20.945679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:20.945686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:20.947522 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.947821 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.947823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.947824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.947875 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:20.948085 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:20.948085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:20.948085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:20.948209 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:20.948273 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:20.948373 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 03:31:20.948521 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260929 03:31:20.949213 29407 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
Entry for principal kudu/127.6.116.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab.
I20260929 03:31:20.949611 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.949479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42598)
0929 03:31:20.949533 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.949582 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 03:31:20.949720 29355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
I20260929 03:31:20.950248 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.950111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42600)
0929 03:31:20.950190 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:20.950216 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 03:31:20.950321 6610 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFuIZmB/build/release/bin/kudu
/tmp/dist-test-taskFuIZmB/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:45419
--webserver_interface=127.6.116.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.116.148:46661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 03:31:20.950357 29407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:45419: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
I20260929 03:31:20.999610 29407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } attempt: 1
I20260929 03:31:21.000133 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:20.999991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42610)
0929 03:31:21.000062 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.000101 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 03:31:21.000494 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.000387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42612)
0929 03:31:21.000434 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.000474 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 03:31:21.000600 29407 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:45419: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
I20260929 03:31:21.001449 29332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } attempt: 3
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:21.008694 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.001758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:32796)
0929 03:31:21.001804 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.001815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.001884 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.002017 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.002018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.002180 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.002245 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.002246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.002903 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.002904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.003523 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.003526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.004018 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.004018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.004107 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.004579 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.004591 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.005804 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.007720 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.007721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.007722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.007895 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.008127 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.008128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.008128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.008182 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.008449 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.008450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.008491 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.008547 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.008630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 03:31:21.008762 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.001821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.6.116.189:43285)
0929 03:31:21.001882 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.001884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.001895 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:31:21.001944 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.001946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.001955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.002012 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.002325 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.002819 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.003626 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.003924 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.004017 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.005907 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.005908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.005909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.005917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.007609 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.007979 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.007981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.007981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.008044 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.008244 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.008244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.008245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.008382 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.008450 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.008591 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.008673 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 03:31:21.009495 29332 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:21.009859 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.009758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42624)
0929 03:31:21.009794 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.009830 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 03:31:21.010192 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.010083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42636)
0929 03:31:21.010145 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.010173 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 03:31:21.010288 29332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:45419: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
W20260929 03:31:21.031273 29414 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 03:31:21.031471 29414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 03:31:21.031488 29414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 03:31:21.031538 29414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 03:31:21.032653 29414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 03:31:21.032692 29414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 03:31:21.032706 29414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 03:31:21.032719 29414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 03:31:21.032768 29414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 03:31:21.034226 29414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.116.148:46661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.116.190:41005,127.6.116.189:43285,127.6.116.188:45419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/krb5kdc/kudu_127.6.116.188.keytab
--principal=kudu/127.6.116.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.116.188:45419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.116.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 01:51:30 UTC on ccycloud.quasar-psbcvq.root.comops.site
build id 4679059
I20260929 03:31:21.034471 29414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 03:31:21.034720 29414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 03:31:21.039358 29420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:21.039409 29414 server_base.cc:1034] running on GCE node
W20260929 03:31:21.039350 29422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 03:31:21.039330 29419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 03:31:21.039745 29414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 03:31:21.040025 29414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 03:31:21.041188 29414 hybrid_clock.cc:648] HybridClock initialized: now 1790652681041178 us; error 42 us; skew 500 ppm
I20260929 03:31:21.041482 29332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } attempt: 1
I20260929 03:31:21.041908 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.041754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42652)
0929 03:31:21.041833 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.041874 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 03:31:21.042454 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.042324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42666)
0929 03:31:21.042376 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.042424 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 03:31:21.042546 29332 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.116.188:45419: Network error: Client connection negotiation failed: client connection to 127.6.116.188:45419: connect: Connection refused (error 111)
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 03:31:21.044888 29414 init.cc:377] Logged in from keytab as kudu/127.6.116.188@KRBTEST.COM (short username kudu)
I20260929 03:31:21.045331 29414 webserver.cc:460] Webserver started at http://127.6.116.188:35357/ using document root <none> and password file <none>
I20260929 03:31:21.045570 29414 fs_manager.cc:359] Metadata directory not provided
I20260929 03:31:21.045615 29414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 03:31:21.045781 29414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 03:31:21.046845 29414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data/instance:
uuid: "c2d1ac1f334741ee98a4108bc7213398"
format_stamp: "Formatted at 2026-09-29 03:31:21 on dist-test-slave-z2zg"
server_key: "c903bf38afcefc81e61284aa5d414216"
server_key_iv: "ebed12635e92880db21ebc0f3e82f6c6"
server_key_version: "encryptionkey@0"
I20260929 03:31:21.047247 29414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal/instance:
uuid: "c2d1ac1f334741ee98a4108bc7213398"
format_stamp: "Formatted at 2026-09-29 03:31:21 on dist-test-slave-z2zg"
server_key: "c903bf38afcefc81e61284aa5d414216"
server_key_iv: "ebed12635e92880db21ebc0f3e82f6c6"
server_key_version: "encryptionkey@0"
I20260929 03:31:21.048604 29414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:21.049361 29429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 03:31:21.049566 29414 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 03:31:21.049638 29414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data,/tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal
uuid: "c2d1ac1f334741ee98a4108bc7213398"
format_stamp: "Formatted at 2026-09-29 03:31:21 on dist-test-slave-z2zg"
server_key: "c903bf38afcefc81e61284aa5d414216"
server_key_iv: "ebed12635e92880db21ebc0f3e82f6c6"
server_key_version: "encryptionkey@0"
I20260929 03:31:21.049721 29414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 03:31:21.062048 29414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 03:31:21.063879 29414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 03:31:21.064025 29414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 03:31:21.067802 29414 rpc_server.cc:307] RPC server started. Bound to: 127.6.116.188:45419
I20260929 03:31:21.067857 29481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.116.188:45419 every 8 connection(s)
I20260929 03:31:21.068224 29414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/data/info.pb
I20260929 03:31:21.068727 29407 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } attempt: 2
I20260929 03:31:21.069382 29482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 03:31:21.071199 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.189@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:21.074769 6610 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFuIZmB/build/release/bin/kudu as pid 29414
I20260929 03:31:21.074880 6610 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFuIZmB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790652420563207-6610-0/minicluster-data/master-2/wal/instance
I20260929 03:31:21.075109 6610 external_mini_cluster.cc:1428] Setting key e329951285e4d6abcc38ae80776b683c
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:21.078012 29484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.069197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.6.116.188:45419)
0929 03:31:21.069348 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.069357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.069379 (+ 22us) server_negotiation.cc:407] Connection header received
0929 03:31:21.069854 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.069860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.069905 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.070012 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.070366 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.071308 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.072286 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.072679 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.072797 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.074470 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.074484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.074484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.074495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.076317 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.076726 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.076729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.076729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.076834 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.077060 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.077061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.077062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.077386 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.077464 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.077680 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.077799 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260929 03:31:21.078099 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.069056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42678)
0929 03:31:21.069122 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.069133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.069191 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.070031 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.070033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.070207 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.070260 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.070264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.071395 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.071396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.072213 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.072215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.072762 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.072763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.072817 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.073229 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.073234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.074383 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.076431 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.076432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.076432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.076640 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.076920 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.076921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.076921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.076983 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.077468 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.077469 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.077521 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.077605 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.077983 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 03:31:21.079763 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.071814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.6.116.190:41005)
0929 03:31:21.071860 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.071861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.072058 (+ 197us) server_negotiation.cc:407] Connection header received
0929 03:31:21.072104 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.072104 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.072112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.072164 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.072636 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.073214 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.073988 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.074238 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.074297 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.076787 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.076789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.076789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.076796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.078483 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.078938 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.078939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.078940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.078997 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.079222 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.079223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.079223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.079340 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.079418 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.079600 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.079674 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 03:31:21.079779 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.071770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.190:41005 (local address 127.0.0.1:45928)
0929 03:31:21.071952 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.071995 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.072055 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.072180 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.072182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.072478 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.072535 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.072537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.073320 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.073321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.073884 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.073886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.074335 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.074337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.074402 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.074953 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.074958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.076719 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.078571 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.078579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.078581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.078853 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.079096 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.079096 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.079097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.079158 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.079411 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.079418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.079467 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.079557 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.079675 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260929 03:31:21.079928 29407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: Bootstrap starting.
I20260929 03:31:21.080468 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } has no permanent_uuid. Determining permanent_uuid...
I20260929 03:31:21.080686 29407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:21.081034 29407 log.cc:824] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:21.082516 29407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: No bootstrap required, opened a new log
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:21.084895 29407 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.085073 29407 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08d6fbc2426422eb9d6faaf4d515287, State: Initialized, Role: FOLLOWER
I20260929 03:31:21.085175 29407 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.085506 29407 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:21.085454 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.085934 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.086057 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.075608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.6.116.188:45419)
0929 03:31:21.075739 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.075741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.076504 (+ 763us) server_negotiation.cc:407] Connection header received
0929 03:31:21.076570 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.076571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.076581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.076640 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.077396 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.078084 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.079320 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.079601 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.079666 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.082336 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.082338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.082338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.082345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.084487 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.084866 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.084867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.084867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.084919 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.085146 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.085147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.085147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.085267 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.085406 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.085814 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.085888 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260929 03:31:21.087548 29500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:21.087603 29500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:21.087762 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.080730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.189:43285 (local address 127.0.0.1:32808)
0929 03:31:21.080768 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.080778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.080813 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.080965 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.080966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.081074 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.081142 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.081144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.081995 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.081996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.082611 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.082614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.082961 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.082962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.083030 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.083503 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.083508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.084711 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.086584 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.086585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.086586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.086807 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.087014 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.087015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.087016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.087114 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.087446 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.087446 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.087489 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.087559 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.087648 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 03:31:21.088304 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.080772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.6.116.189:43285)
0929 03:31:21.080811 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.080813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.080821 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:21.080866 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.080867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.080876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.080904 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.081203 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.081903 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.082703 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.082885 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.082962 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.084797 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.084799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.084799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.084806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.086486 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.086868 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.086869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.086870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.086932 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.087203 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.087204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.087205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.087354 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.087417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.088087 (+ 670us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.088181 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:31:21.088215 29339 master_runner.cc:186] Error getting master registration for 127.6.116.190:41005: OK, instance_id { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" instance_seqno: 1790652680775695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:31:21.090777 29339 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:31:21.090904 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.190@KRBTEST.COM
I20260929 03:31:21.095652 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.087134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.6.116.190:41005)
0929 03:31:21.087172 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.087174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.087183 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:31:21.087206 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.087207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.087215 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.087267 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.087896 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.088578 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.089490 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.089778 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.089853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.092301 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.092303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.092303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.092310 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.093981 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.094364 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.094365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.094365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.094681 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.094894 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.094895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.094896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.095016 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.095215 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.095405 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.095482 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652681, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.188@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:21.098815 29485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.091184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42704)
0929 03:31:21.091211 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.091221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.091260 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.091396 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.091397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.091538 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.091564 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.091565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.092428 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.092429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.093742 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.093745 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.094232 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.094233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.094294 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.094678 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.094682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.096075 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.097861 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.097862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.097863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.098022 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.098231 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.098232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.098232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.098284 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.098522 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.098523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.098559 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.098632 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.098728 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260929 03:31:21.099330 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.091216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.6.116.188:45419)
0929 03:31:21.091247 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.091249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.091257 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:21.091315 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.091315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.091324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.091366 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.091654 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.092344 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.093837 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.094149 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.094224 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.096144 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.096144 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.096145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.096151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.097769 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.098105 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.098106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.098106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.098167 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.098354 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.098355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.098355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.098458 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.098529 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.098657 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.099270 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":532,"server-negotiator.queue_time_us":19}
I20260929 03:31:21.100481 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: Bootstrap starting.
I20260929 03:31:21.101063 29482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:21.101358 29482 log.cc:824] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: Log is configured to *not* fsync() on all Append() calls
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.116.189@KRBTEST.COM
I20260929 03:31:21.102067 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: No bootstrap required, opened a new log
I20260929 03:31:21.103751 29482 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.103870 29482 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2d1ac1f334741ee98a4108bc7213398, State: Initialized, Role: FOLLOWER
I20260929 03:31:21.103978 29482 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.104249 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.104310 29482 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:21.104333 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.104899 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.096358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.6.116.189:43285)
0929 03:31:21.096397 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.096399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.096408 (+ 9us) server_negotiation.cc:407] Connection header received
0929 03:31:21.096430 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.096431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.096442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.096539 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.097155 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.097825 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.098763 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.099015 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.099167 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.101888 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.101890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.101890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.101897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.103507 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.103914 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.103915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.103915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.103982 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.104216 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.104217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.104217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.104417 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.104534 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.104737 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.104813 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:31:21.106336 29514 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:21.106434 29514 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:21.106657 6610 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 03:31:21.107558 29306 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "07b582cdcd52440f8c0b13606d8442e9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45944
I20260929 03:31:21.106494 29414 master_runner.cc:186] Error getting master registration for 127.6.116.190:41005: OK, instance_id { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" instance_seqno: 1790652680775695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 03:31:21.108731 29414 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 03:31:21.117686 29332 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } attempt: 2
Sep 29 03:31:21 dist-test-slave-z2zg krb5kdc[29251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790652680, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.116.190@KRBTEST.COM for kudu/127.6.116.188@KRBTEST.COM
I20260929 03:31:21.124866 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.117965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.116.188:45419 (local address 127.0.0.1:42720)
0929 03:31:21.118013 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 03:31:21.118023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 03:31:21.118059 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 03:31:21.118256 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 03:31:21.118257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 03:31:21.118409 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 03:31:21.118457 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.118459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.119395 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.119396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.119878 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 03:31:21.119879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.120319 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.120320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 03:31:21.120378 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.120773 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 03:31:21.120779 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 03:31:21.122112 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 03:31:21.123779 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.123780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.123780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.123992 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.124202 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 03:31:21.124203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 03:31:21.124204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 03:31:21.124256 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 03:31:21.124547 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 03:31:21.124548 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 03:31:21.124588 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 03:31:21.124677 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 03:31:21.124774 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 03:31:21.124866 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.118135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.6.116.188:45419)
0929 03:31:21.118169 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.118171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.118177 (+ 6us) server_negotiation.cc:407] Connection header received
0929 03:31:21.118194 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.118194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.118203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.118252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.118550 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.119285 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.119964 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.120262 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.120321 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.122213 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.122214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.122215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.122221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.123668 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.124075 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.124076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.124076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.124148 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.124331 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.124333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.124333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.124455 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.124541 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.124714 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.124781 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:31:21.126644 29332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: Bootstrap starting.
I20260929 03:31:21.127346 29332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: Neither blocks nor log segments found. Creating new log.
I20260929 03:31:21.127647 29332 log.cc:824] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: Log is configured to *not* fsync() on all Append() calls
I20260929 03:31:21.128433 29332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: No bootstrap required, opened a new log
I20260929 03:31:21.129748 29332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.129878 29332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b582cdcd52440f8c0b13606d8442e9, State: Initialized, Role: FOLLOWER
I20260929 03:31:21.129964 29332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.130260 29332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 03:31:21.130309 29306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 03:31:21.130350 29306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:21.131301 29306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
I20260929 03:31:21.131558 29306 leader_election.cc:288] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [CANDIDATE]: Term 1 election: Requested vote from peers a08d6fbc2426422eb9d6faaf4d515287 (127.6.116.189:43285), c2d1ac1f334741ee98a4108bc7213398 (127.6.116.188:45419)
I20260929 03:31:21.132023 29381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07b582cdcd52440f8c0b13606d8442e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a08d6fbc2426422eb9d6faaf4d515287"
I20260929 03:31:21.132149 29381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:21.133052 29381 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b582cdcd52440f8c0b13606d8442e9 in term 1.
I20260929 03:31:21.133307 29456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07b582cdcd52440f8c0b13606d8442e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2d1ac1f334741ee98a4108bc7213398"
I20260929 03:31:21.133399 29456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [term 0 FOLLOWER]: Advancing to term 1
I20260929 03:31:21.134051 29456 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07b582cdcd52440f8c0b13606d8442e9 in term 1.
I20260929 03:31:21.134316 29280 leader_election.cc:302] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b582cdcd52440f8c0b13606d8442e9, c2d1ac1f334741ee98a4108bc7213398; no voters:
I20260929 03:31:21.134557 29533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 03:31:21.130215 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.134869 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.134768 29533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [term 1 LEADER]: Becoming Leader. State: Replica: 07b582cdcd52440f8c0b13606d8442e9, State: Running, Role: LEADER
I20260929 03:31:21.135197 29533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } }
W20260929 03:31:21.137012 29528 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 03:31:21.137059 29528 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 03:31:21.137396 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b582cdcd52440f8c0b13606d8442e9. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.137626 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:21.146406 29335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.135902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.6.116.190:41005)
0929 03:31:21.135948 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.135950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.138703 (+ 2753us) server_negotiation.cc:407] Connection header received
0929 03:31:21.138788 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.138789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.138799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.138860 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.139648 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.140342 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.141357 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.141655 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.141701 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.143276 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.143278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.143278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.143285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.145125 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.145546 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.145548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.145548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.145605 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.145835 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.145836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.145836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.145956 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.146025 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.146227 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.146320 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 03:31:21.146737 29540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 03:31:21.147105 29540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 03:31:21.148422 29381 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [term 1 FOLLOWER]: Refusing update from remote peer 07b582cdcd52440f8c0b13606d8442e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:31:21.148705 29456 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [term 1 FOLLOWER]: Refusing update from remote peer 07b582cdcd52440f8c0b13606d8442e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 03:31:21.148854 29517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:31:21.149240 29517 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 03:31:21.150390 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b582cdcd52440f8c0b13606d8442e9. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.150485 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.151480 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07b582cdcd52440f8c0b13606d8442e9. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.151607 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.151867 29517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.151945 29517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:21.152447 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.152532 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.152611 29541 mvcc.cc:205] Tried to move back new op lower bound from 7334513381981794304 to 7334513381930668032. Current Snapshot: MvccSnapshot[applied={T|T < 7334513381981794304}]
I20260929 03:31:21.155535 29542 mvcc.cc:205] Tried to move back new op lower bound from 7334513381981794304 to 7334513381930668032. Current Snapshot: MvccSnapshot[applied={T|T < 7334513381981794304}]
I20260929 03:31:21.156515 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.136258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.6.116.189:43285)
0929 03:31:21.136407 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.136409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.139186 (+ 2777us) server_negotiation.cc:407] Connection header received
0929 03:31:21.139283 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.139284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.139293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.139321 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.140911 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.141645 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.148324 (+ 6679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.148612 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.148657 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.151512 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.151513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.151513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.151528 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.153415 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.154909 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.154910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.154910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.154968 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.155482 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.155483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.155484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.155615 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.155699 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.156342 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.156425 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 03:31:21.157110 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:21.136439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.6.116.188:45419)
0929 03:31:21.136492 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.136494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.140103 (+ 3609us) server_negotiation.cc:407] Connection header received
0929 03:31:21.140190 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.140191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.140201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.140228 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.146594 (+ 6366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.147323 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.149002 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:21.149286 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:21.149328 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:21.151855 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 03:31:21.151857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 03:31:21.151857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 03:31:21.151864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 03:31:21.154234 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.155225 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.155226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.155226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.155278 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 03:31:21.155924 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 03:31:21.155925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 03:31:21.155925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 03:31:21.156055 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 03:31:21.156136 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:21.156937 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 03:31:21.157012 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 03:31:21.159587 29540 catalog_manager.cc:1347] Generated new cluster ID: 1f223d14074946f498c54b84ddafd490
I20260929 03:31:21.159633 29540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 03:31:21.163841 29533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.163918 29533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9 [sys.catalog]: This master's current role is: LEADER
I20260929 03:31:21.164434 29503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07b582cdcd52440f8c0b13606d8442e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b582cdcd52440f8c0b13606d8442e9" member_type: VOTER last_known_addr { host: "127.6.116.190" port: 41005 } } peers { permanent_uuid: "a08d6fbc2426422eb9d6faaf4d515287" member_type: VOTER last_known_addr { host: "127.6.116.189" port: 43285 } } peers { permanent_uuid: "c2d1ac1f334741ee98a4108bc7213398" member_type: VOTER last_known_addr { host: "127.6.116.188" port: 45419 } } }
I20260929 03:31:21.164508 29503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 03:31:21.168751 29544 mvcc.cc:205] Tried to move back new op lower bound from 7334513381981794304 to 7334513381930668032. Current Snapshot: MvccSnapshot[applied={T|T < 7334513381981794304}]
I20260929 03:31:21.187393 29540 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 03:31:21.188161 29540 catalog_manager.cc:1504] Loading token signing keys...
I20260929 03:31:21.196151 29540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07b582cdcd52440f8c0b13606d8442e9: Generated new TSK 0
I20260929 03:31:21.196346 29540 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 03:31:21.219558 29335 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.217718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.6.116.190:41005)
0929 03:31:21.217767 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.217769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.218678 (+ 909us) server_negotiation.cc:407] Connection header received
0929 03:31:21.218918 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.218919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.218933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.218970 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.219487 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45954: BlockingRecv error: recv got EOF from 127.0.0.1:45954 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:31:21.220366 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.218178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32816 (local address 127.6.116.189:43285)
0929 03:31:21.218221 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.218223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.219816 (+ 1593us) server_negotiation.cc:407] Connection header received
0929 03:31:21.219924 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.219925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.219937 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.219971 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.220306 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32816: BlockingRecv error: recv got EOF from 127.0.0.1:32816 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 03:31:21.221164 29487 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.218526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.6.116.188:45419)
0929 03:31:21.218577 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.218579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.220595 (+ 2016us) server_negotiation.cc:407] Connection header received
0929 03:31:21.220732 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.220733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.220745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.220778 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.221105 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42736: BlockingRecv error: recv got EOF from 127.0.0.1:42736 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 03:31:21.223698 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.222453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.6.116.189:43285)
0929 03:31:21.222491 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.222493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.223100 (+ 607us) server_negotiation.cc:407] Connection header received
0929 03:31:21.223244 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.223245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.223263 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.223296 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.223654 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32828: BlockingRecv error: recv got EOF from 127.0.0.1:32828 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:31:21.226818 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.226024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.6.116.189:43285)
0929 03:31:21.226060 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.226061 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.226287 (+ 226us) server_negotiation.cc:407] Connection header received
0929 03:31:21.226404 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.226405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.226416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.226449 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.226774 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32836: BlockingRecv error: recv got EOF from 127.0.0.1:32836 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 03:31:21.230777 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.229998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.6.116.189:43285)
0929 03:31:21.230034 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.230035 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.230247 (+ 212us) server_negotiation.cc:407] Connection header received
0929 03:31:21.230351 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.230351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.230362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.230409 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.230734 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32846: BlockingRecv error: recv got EOF from 127.0.0.1:32846 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 03:31:21.237003 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.236186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.6.116.189:43285)
0929 03:31:21.236222 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.236224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.236472 (+ 248us) server_negotiation.cc:407] Connection header received
0929 03:31:21.236579 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.236581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.236591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.236624 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.236956 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32850: BlockingRecv error: recv got EOF from 127.0.0.1:32850 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 03:31:21.247190 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.246292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.6.116.189:43285)
0929 03:31:21.246342 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.246344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.246605 (+ 261us) server_negotiation.cc:407] Connection header received
0929 03:31:21.246713 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.246714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.246726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.246760 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.247142 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32860: BlockingRecv error: recv got EOF from 127.0.0.1:32860 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 03:31:21.265494 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.264549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.6.116.189:43285)
0929 03:31:21.264594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.264596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.264854 (+ 258us) server_negotiation.cc:407] Connection header received
0929 03:31:21.264974 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.264975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.264987 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.265023 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.265436 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32872: BlockingRecv error: recv got EOF from 127.0.0.1:32872 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 03:31:21.299696 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.298864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.6.116.189:43285)
0929 03:31:21.298918 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.298920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.299090 (+ 170us) server_negotiation.cc:407] Connection header received
0929 03:31:21.299186 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.299187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.299199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.299275 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.299619 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32874: BlockingRecv error: recv got EOF from 127.0.0.1:32874 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260929 03:31:21.365504 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.364833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.6.116.189:43285)
0929 03:31:21.364931 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.364933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.365012 (+ 79us) server_negotiation.cc:407] Connection header received
0929 03:31:21.365062 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.365063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.365073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.365138 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.365434 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32882: BlockingRecv error: recv got EOF from 127.0.0.1:32882 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 03:31:21.495335 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.494624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.6.116.189:43285)
0929 03:31:21.494726 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.494728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.494771 (+ 43us) server_negotiation.cc:407] Connection header received
0929 03:31:21.494818 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.494819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.494829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.494882 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.495274 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32886: BlockingRecv error: recv got EOF from 127.0.0.1:32886 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 03:31:21.753328 29412 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 03:31:21.752562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.6.116.189:43285)
0929 03:31:21.752638 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:21.752640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:21.752781 (+ 141us) server_negotiation.cc:407] Connection header received
0929 03:31:21.752820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:21.752821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:21.752835 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:21.752903 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 03:31:21.753263 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32898: BlockingRecv error: recv got EOF from 127.0.0.1:32898 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 03:31:22.087993 29500 catalog_manager.cc:1259] Loaded cluster ID: 1f223d14074946f498c54b84ddafd490
I20260929 03:31:22.088059 29500 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: loading cluster ID for follower catalog manager: success
I20260929 03:31:22.088783 29500 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: acquiring CA information for follower catalog manager: success
I20260929 03:31:22.089004 29500 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a08d6fbc2426422eb9d6faaf4d515287: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:31:22.106894 29514 catalog_manager.cc:1259] Loaded cluster ID: 1f223d14074946f498c54b84ddafd490
I20260929 03:31:22.106951 29514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: loading cluster ID for follower catalog manager: success
I20260929 03:31:22.107800 29514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: acquiring CA information for follower catalog manager: success
I20260929 03:31:22.108029 29514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2d1ac1f334741ee98a4108bc7213398: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 03:31:22.268833 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:22.266432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.6.116.189:43285)
0929 03:31:22.266552 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:22.266554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:22.266565 (+ 11us) server_negotiation.cc:407] Connection header received
0929 03:31:22.266633 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:22.266634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:22.266684 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:22.266756 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:22.266885 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.267396 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.268119 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.268301 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.268349 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:22.268487 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:22.268603 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:22.268662 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:22.268703 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 03:31:22.268719 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260929 03:31:22.272746 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:22.270596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.6.116.188:45419)
0929 03:31:22.270708 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:22.270710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:22.270718 (+ 8us) server_negotiation.cc:407] Connection header received
0929 03:31:22.270788 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:22.270789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:22.270821 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:22.270887 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:22.270980 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.271497 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.272148 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.272334 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.272383 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:22.272500 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:22.272589 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:22.272641 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:22.272676 (+ 35us) server_negotiation.cc:299] Negotiation successful
0929 03:31:22.272692 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260929 03:31:22.277370 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:22.274281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.6.116.189:43285)
0929 03:31:22.274323 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:22.274325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:22.274566 (+ 241us) server_negotiation.cc:407] Connection header received
0929 03:31:22.274676 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:22.274677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:22.274687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:22.274735 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:22.274952 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.275724 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.276586 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.276875 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.276955 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:22.277090 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:22.277143 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:22.277197 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:22.277291 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 03:31:22.277310 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 03:31:22.277372 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:22.274319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.6.116.190:41005)
0929 03:31:22.274437 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:22.274439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:22.274506 (+ 67us) server_negotiation.cc:407] Connection header received
0929 03:31:22.274593 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:22.274595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:22.274640 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:22.274715 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:22.274806 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.275428 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.276334 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.276646 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.276736 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:22.276875 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:22.276963 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:22.277026 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:22.277262 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 03:31:22.277292 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260929 03:31:22.277523 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 03:31:22.274577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42752 (local address 127.6.116.188:45419)
0929 03:31:22.274610 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 03:31:22.274612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 03:31:22.274869 (+ 257us) server_negotiation.cc:407] Connection header received
0929 03:31:22.274895 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 03:31:22.274896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 03:31:22.274906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 03:31:22.274959 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 03:31:22.275119 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.275790 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.276694 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 03:31:22.276960 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 03:31:22.277024 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 03:31:22.277161 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 03:31:22.277190 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 03:31:22.277240 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 03:31:22.277423 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 03:31:22.277442 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 03:31:22.278968 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 29264
I20260929 03:31:22.284152 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 29339
I20260929 03:31:22.288942 6610 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFuIZmB/build/release/bin/kudu with pid 29414
2026-09-29T03:31:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2792 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5574 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (261732 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST